Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

backend python code #4

Open
wants to merge 20 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 14 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,2 +1 @@
NTHUOJ_backend
==============
python code is in the ojbackend directory
2 changes: 2 additions & 0 deletions ojbackend/ReadMe
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
1. execute install.py to set up machine.config and ojdatase.config
2. execute dispatcher.py as daemon and it will check if there are new submissions need to be judged
78 changes: 78 additions & 0 deletions ojbackend/dispatcher.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
"""
dispatcher.py
This process continuously checks whether there exist unjudged submissions.
"""

from dispatcherFunc import *
import time
import logging
import subprocess, sys
logging.basicConfig(filename = 'dispatcher.log', level = logging.INFO, format = '%(asctime)s ::%(message)s',datafmt = '%m/%d/%Y %I:%M:%S %p')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One coding convention is at most 80 characters in one line,
thanks~

logging.info('==========Dispatcher Started==========')

machineInfo = getMachine()
if machineInfo != None:
initMachine(machineInfo)
logging.info('getMachine Success')
else:
logging.info('getMachine Error')
logging.info('Please check the file machineInfo.config to check the settings')
logging.info('==========Dispatcher Finished==========')
exit(0)

dbIP, dbUser, dbPasswd, dbName = getdbInfo()
if dbIP == None or dbUser == None or dbPasswd == None or dbName == None:
logging.info('getdbInfo Error')
logging.info('Please check the settings in ojdatabase.config')
logging.info('==========Dispatcher Finished==========')
exit(0)
logging.info('connect to database')

while True:

DB = connectDB(dbIP, dbUser, dbPasswd, dbName)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do not connect DB every time

if DB == None:
logging.info('connect database error')
logging.info('Please check the settings in ojdatabase.config')
logging.info('==========Dispatcher Finished==========')
exit(0)
cur = DB.cursor()
cur.execute("USE nthuoj;")
cur.execute("SELECT * FROM problem_submission WHERE status = 'WAIT' ORDER BY id ASC LIMIT 100;")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Prefer to take SQL into one variable

subsmissionStat = "
SELECT * FROM problem_submission
WHERE status = 'WAIT'
ORDER BY id ASC
LIMIT 100;
cur.execute(submissionStat)
"

Will be easier to maintain in the future

sidQuery = cur.fetchone()
sid = 1
pid = 2
judgeLanguage = 'cpp'
judgeURL = 'judgeURL'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we add a function to choose language and URL?

if sidQuery != None:
sid = sidQuery[0]
pid = sidQuery[1]
logging.info('sidQuery Success!')
logging.info('sid = %d, pid = %d' % (sid, pid))
cur.execute("SELECT * FROM problem_problem where id = '%d';" % pid)
pidQuery = cur.fetchone()
judgeSource = pidQuery[11]
judgeType = pidQuery[12]
judgeLanguage = pidQuery[13]
logging.info('pidQuery Success!')
logging.info('judgeSource = %s, judgeType = %s, judgeLanguage = %s' % (judgeSource, judgeType, judgeLanguage))

if judgeSource == "LOCAL":
idleMachine = None
logging.info('get idleMachine')
while(idleMachine == None):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

change to while idleManchine == None

idleMachine = getIdleMachine()
time.sleep(1)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

time.sleep(1) should be front of idleMachine = getIdleMachine()

logging.info('idleMachine = %s' % idleMachine)
cur.execute("UPDATE problem_submission SET status = 'JUDGING' WHERE id = '%d';" % sid)
DB.commit()
judgeIP = machineInfo[idleMachine]
judgeURL = judgeIP + "/interface.py"
logging.info('get judgeURL = %s' % judgeURL)
else :
cur.execute("UPDATE problem_submission SET status = 'JUDGING' WHERE id = '%d';" % sid)
logging.info('send info to other judge')
"""
to be continued
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please add a simple description to help other know what function will be implemented here

"""
time.sleep(1)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One suggestion for this file is to use try-catch-finally statement.
We can force close connection of DB at the end of process and DB will not have too many connections and simply handle some exception not just rely on log.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change if to for loop not just deal with one submission

76 changes: 76 additions & 0 deletions ojbackend/dispatcherFunc.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
"""
dispatcherFunc.py
This file provides related functions for dispatcher.php.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you mean dispatcher.py?

"""
from bash import bash
import MySQLdb


def getMachine():
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Want to discuss this implement.
How about loading config when dispatcher starts and sync the configuration periodically?
Implement singleton for configuration .

#load judge information from judge.config
#the information include judge name, ip address
machineInfo = {}
with open("machineInfo.config",'r') as f:
try:
for line in f:
(machine, ip) = line.split()
machineInfo[machine] = ip
return machineInfo
except:
print "encounter troubles when opening machineInfo.config"
return None
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

two blank lines between functions

def initMachine(machineInfo):
#create a dir and a file for machineStatus
#0 means availabe, 1 means occupied
try:
bash('test -d machineStatusDir && rm -r machineStatusDir')
bash('mkdir machineStatusDir')
with open("machineStatusDir/machineStatus.config",'w') as f:
for machineName in machineInfo.keys():
f.write(machineName + " 0\n")
except:
print "encounter troubles when rm machineStatusDir or mkdir machineStatusDir"
return None
def getIdleMachine():
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto

#to get the machine which is available for judging
machinePosition = 0
IdleMachine = ''
try:
with open("machineStatusDir/machineStatus.config",'r+') as f:
for line in f:
if(line.split()[1] == '0'):
machineName = line.split()[0]
f.seek(machinePosition + len(machineName) + 1, 0)
f.write('1')
IdleMachine = machineName
return IdleMachine
break
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why you need return and break

machinePosition = machinePosition + len(line)
return None
except:
return None
def getdbInfo():
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto

with open("ojdatabase.config",'r') as f:
try:
for line in f:
if(line.split()[0] == "host"):
dbIP = line.split()[2]
elif(line.split()[0] == "user"):
dbUser = line.split()[2]
elif(line.split()[0] == "passwd"):
dbPasswd = line.split()[2]
elif(line.split()[0] == "db"):
dbName = line.split()[2]
except:
print "Log dbInfo Error\nPlease check ojdatabase.config"
return None, None, None, None
return dbIP, dbUser, dbPasswd, dbName
def connectDB(dbIP, dbUser, dbPasswd, dbName):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto

try:
DB = MySQLdb.connect(host = dbIP, user = dbUser, passwd = dbPasswd, db = dbName)
print "connect db success\n"
return DB
except:
print "connect DB error\n"
return None

40 changes: 40 additions & 0 deletions ojbackend/install.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
"""
install.py
This file let users to set up machineInfo.config and ojdatabase.config file
"""

from bash import bash
def set_MachineInfo():

bash('test -d machineInfo.config && rm -r machineInfo.config')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not use Popen or check_call?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just curious about this, not really a suggestion

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be better if I use Popen or check_call?... I didn't consider too much then.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using Popen can handle different messages by yourself.
On the other way, using check_call will notify the user when procedure call fails.
But I'm not sure whether bash can get the similar results.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it!

num = raw_input("Please input the total number of the judges:\n")
if num > 0:
try:
with open("machineInfo.config",'w') as f:
for i in range(0,int(num)):
judgeName = raw_input("Please input the judge's name:\n")
judgeIP = raw_input("Please input the judge's ip:\n")
f.write(judgeName + " " + judgeIP + "\n")
except:
print "Error occurs when setting machineInfo.config\n"
exit(0)

def set_OjDatabase():
bash('test -d ojdatabase.config && rm -r ojdatabase.config')
try:
with open("ojdatabase.config",'w') as f:
host = raw_input("Please input the ip of the database:\n")
user = raw_input("Please input the username:\n")
passwd = raw_input("Please input the password:\n")
db = raw_input("Please input the name of the database:\n")
f.write("host = " + host + "\n")
f.write("user = " + user + "\n")
f.write("passwd = " + passwd + "\n")
f.write("db = " + db + "\n")
except:
print "Error occurs when setting ojdatabase.config\n"
exit(0)

set_OjDatabase()
set_MachineInfo()