Latest Screenshots

Redis

Tonight I was testing the recommendation engine and I noticed that it was recording the questions that the user missed but was not generating any recommendations.

I found out that I need to have a server install that would process the "jobs" of calculating the recommendations. Apparently this is something that is very math intensive and its better to do it this way. Plus its completely compatible with heroku.

I did the following

1. I installed redis-server and configured it to listen to incoming connections.
2. Installed Resque to manage the queue of jobs and send them to redis.

After that I just ran

<pre><code>
QUEUE=* rake environment resque:work
</code></pre>

And now everything works like magic.

Recommendation Engine

Building the recommendation engine was easy, it took me around 20 minutes.

I installed and configured the recommendable gem (https://github.com/davidcelis/recommendable). After that I just had to add one line of code in the stats generation logic that makes the user like a question if its marked as wrong.

Now I can recommend questions to a user based on what other users have missed.

>> user.recommended_questions
=> [#<question_id: 88="88">, #<question_id: 200="200">, #<question_id: 50="50">...]

The next step is to make it work for concept, concept groups, and categories.

Stats retrieval

Assignments can be done more than once. Every time a user submits an assignment new stats are generated. This becomes an issue when pulling stats for an assignment because there will be duplicate stats.

To fix this issue I wrote a method that pulls the latest set of stats for an assignment.

  

def self.latest_for_assignment assignment
    categories = []
    concept_groups = []
    concepts = []
    Category.all.each do |c|
      categories << Stat.where(:statable_id => c.id, :statable_type => c.class.name ,:assignment_id => assignment.id  ).last
    end
    ConceptGroup.all.each do |c|
      concept_groups << Stat.where(:statable_id => c.id, :statable_type => c.class.name, :assignment_id => assignment.id).last
    end
    Concept.all.each do |c|
      concepts << Stat.where(:statable_id => c.id, :statable_type => c.class.name, :assignment_id => assignment.id ).last
    end
    [categories,concept_groups,concepts]
  end

I also change the way the show method (in the controller) works.

def show
  @categories,@concept_groups,@concepts = Stat.latest_for_assignment @assignment
  @categories.compact!
  @concept_groups.compact!
  @concepts.compact!
end

The compact! clears nils from the arrays 

Stats Page

Stats are being generated correctly. Here is a preview.

Stats generation refactoring

Most of the stats generation methods had unnecessary db calls and overall were very messy. Last week I began rewriting these methods.

Last week I was also able to compile a small group of questions with all their categorization models so this week i'm beginning to do the recommendation engine.

I saw this project and I think I might be able to use it if I modify the rating system to fit questions.
https://github.com/davidcelis/recommendable


Next week I will work on this, continue to make small refactoring, and clean up the UI.

Refactoring

This week I am doing heavy refactoring on the stats generation and after that I will work on making the UI look better.

I will post the changes I made to the code and explain why.

Weekly Update

The Assignment taking part of the system is done and it is working with backbone so it looks cool.

Beta testing wont be able to start with classrooms because we don't have enough content digitized yet. However the system works and should be able to accept properly formatted content.

The next step from here is polishing aspects on the tutoring side so that tutors have a better view of how a class/student is doing.
I'm thinking about doing a separate user model for tutors since that way I could separate the controllers and views while still retaining a RESTful functionality.

El Beta

Today I finished most of the views that are going to be used in the beta.
The beta is going to be the following:

1. A user signs_up / signs_in
2. Assignment list is displayed
3. User can take assignment or see stats from a previous one
4. User can see stats after he takes an assignment

Backbone.js

I finished the UI for taking an exam. The Backbone code looks like this

  questions = new Altair.Collections.QuestionsCollection
  assignments = new Altair.Collections.AssignmentsCollection
  questions.reset(#{@assignment.questions.to_json})
  assignments.reset(#{@assignment.to_json})
  assignment = assignments.get(1)
  answer = new Altair.Collections.AnswersCollection

  $(function() {
  render_question()

  function bind_click(){
  $('.anslink').click( function() {
    ans = $(this).text()
    question = $(this).parent().find(".question_id").val()
    answer.create({'question_id' : question, 'answer' : ans})
   })
   }

  function bind_submit(){
  $('.nextquestion').click( function() {
    assignment.fetch()
    assignment.set({'current_id' : current + 1 })
    assignment.save()
    render_question()
   })
   }

   function render_question(){
    assignment.fetch()
    current = assignment.get('current_id')
    q = questions.get(current)
    r = new Altair.Views.Questions.ShowView({'model': q})
    r.render()
    $('#question').html(r.$el.html())
    rightWrong()
    bind_click()
    bind_submit()
   }
}

Backbone.js

This week I'm working with Backbone.js to make a new UI experience when the user "takes" an assignment.

Changes to Exam Model

Yesterday we decided to change how the exams work on the system. It used to be that every exam has a user_id and references a user. Now Exams are only lists of questions and can be referenced to any user. A new joint model called assignment will reference a user and an exam.

Subscription App

One of the many tedious aspects of the tutoring business is to schedule the groups of students.

Last week I developed a subscription system that collects several data points on the user and then groups them by ensuring compatibility with other users. If a user is not compatible with any of the current groups the system will create a new one.

I tried to make this app as modular as possible. Each data point collected is actually a model and is unrelated to the other models. Each model has a method called compatible?( model ). The parameter it takes is another model of its own class. It then compares the models and returns true if they are compatible. Comparing two users compares all their sub models for compatibility. Comparing a user to a group compares that user to all the users in the group.

If we want to add more data then we just need to add another model, write its compatibility method, and make the user compatibility check use it.

Tutor Interface

This week I spent some time redesigning the tutor side of system.

Nothing major, just making it cleaner.

Vocab App

Last week I developed an API to enable a client to connect to the system and pull a list of words and their definitions. The client program then runs a game and saves the results in the system.

I also wrote some ruby scripts to input the words and definitions into the system, also using the api.

require 'net/http'
require 'json'
require 'csv'

class Poster

  def initialize
  @words = URI('http://URL GOES HERE/words.json')
  @definitions = URI('http://URL GOES HERE/definitions.json')
  end

  def post_word name, difficulty
    res = Net::HTTP.post_form(@words,'word[name]' => name.encode('UTF-8'), 'word[difficulty]' => difficulty.encode('UTF-8'))
    JSON.parse(res.body)['id']
  end

  def post_definition word, body
    res = Net::HTTP.post_form(@definitions,'definition[word_id]' => word, 'definition[body]' => body.encode('UTF-8'))
    puts res.body
  end


end
poster = Poster.new

CSV.foreach(ARGV[0], {:headers => true, :header_converters => :symbol, :col_sep => "\t" }) do |row|
  poster.post_definition(poster.post_word(row[1],row[0]),row[2])
end

The Program goes through a CSV file that has the words, their definitions and their difficulty. It posts a word to the system and gets back a word_id, then it posts the definition passing it the corresponding word_id

Weekend update

This weekend I made several updates to the system.

• created scaffold and api for vocabulary review game

• created a view to export content data (questions and categorization data) data as a CSV file

This took me a while because I could not get the CSV file to be parsed correctly. Question text contained commas so I could not use that as the delimiter. Instead I used tabs. The other problem I had was that there existed \n and \r characters in the question text. To fix this I found some code that effectively cleansed the string. This was the final version:

<%= v.to_s.squeeze(' ').strip.chomp.gsub("\r","").gsub("\n", "")  + "\t" %>



• added a parent method to the stat model which returns the parent of the associated 

• created a way to collect and pull aggregated data from individual users

This week I will be working on the tutor interface.

Content API

Due to the fact that we are a start up company we need to be exploring different sources of traffic and revenue. Because of this we might have a separate group working on game development and mobile applications. However these apps will need to interact with the system to pull data. For example, the apps themselves will not store any content data but rather request it from the system through an API. User data will also be stored through the system. This not only creates a centralized place for data but also enables authorized third party developers to interact with the system without replicating data or making changes to production code.

The API will enable developers to pull question data, create assignments, save user data including answers and generate and pull statistics. All this will be accessed using a standard REST API format.

Tutors and Assignments

Its been a long time since I've looked at the tutor side of the system. This week I created a bunch of methods that together enable the tutor create exams based on certain criteria. For example, a Tutor can now select a category, average difficulty, amount of questions and target user and the system will create and assign an exam.

This is essential to the tutoring side of the business because its going to cut time on the tutor side and enable one tutor to serve more students.

Besides an assignment gun I also need to implement a way for students to be assigned to a tutor. At the moment the users need to be manually added to a Group and groups need to be assigned to a specific tutor. Ideally I would want either a algorithm to auto assign the students to a group based on certain criteria or a front end that will allow the tutors or an admin to assign students to groups and manage tutors.

Abstraction and Polymorphism

Today I was working on the stats generation for the homeworks and I noticed that it took a ridiculous amount of time to pull the aggregated stats for any user.

The problem seemed to be that I had to run a bunch of sql calls in order to get the information to build each Stats block. It seemed unreasonable that I would have to compute this on the fly whenever the user requested it. I cannot loop over all the user's answers every time they want to see their stats.

To solve this I decided to create a new Stat model and use polymorphism to deal with the different elements that would need statistics.

For each question we have several data points that we collect and they have a heirarchy. These are considered to be Statable (polymorphism)
After each exam the the data points from the questions are collected and computed into the "stats". Then a new entry in the Stats table is created with all this information. Here I will take advantage of the queue system in heroku and send these data computing jobs to run there.

Now with this new setup I can pull aggregate data easily by running a simple sql query. More importantly I also now have data for specific time periods. It would have been much harder to see progression over time of a particular user because I would have needed to use the dates on the exams. Now I just get the stats from a range of dates and I can even filter it to specific data points.

The Homework Software

Last semester I started working with a company that runs a tutoring service. I jumped in because they wanted some sort of homework and testing platform that would make it easier to assign and grade homeworks/exams.

I started coding in April, at that point I was just a little bit familiar with rails, heroku and MVC development.

As of now the project has grown tremendously. We are now part of a start up incubator and the goal for me now seems to be to create a software that can be used with any sort of content/curriculum.

We want to create a dead simple homework analytics to help students study smarter, help teachers make their curricula dynamic and keep parents informed about student performance.

My future posts will be regarding this project.

Hello World

Hi, I am a software developer, I write code. In this blog I will be writing about my software projects to keep track of my progress and to share any interesting problems I run into. Hope it helps someone.