Backend Support for Dynamic Custom Fields on Joining Form - Design

[daltonfury42]

Currently when a person joins a queue, he is asked a fixed set of input data, (name and contact number). We want to give the queue owner the ability to define a set of custom set of questions. We need to let the admin set it in a way inspired from the Google Forms model:

We need to design this first.

[daltonfury42]

Proposal - How we define and store questions - JSON

We would have GET/PUT/POST endpoints for each queue that the UI would use to set, edit, and view the questions. Each custom_question would have these fields:

1. Label - Example: "What service are you looking for?"
2. Type: checkbox, file, dropdown, short_text, long_text etc
3. Depending on the type, there would be other fields, like for example, if it's a dropdown or a checkbox, there will be list of choosable options.
4. Repeatable: A boolean field if set to true, would let joiners add another answer. (+ Add more... options)
5. Other fields like, required etc..

An example PUT call to set questions. A hypothetical saloon owner would use this to know what type of haircut, how many people etc:

[
    {
        "type": "SHORT_TEXT",
        "label": "What is your name?"
    },
    {
        "type": "MOBILE_NUMBER",
        "label": "Contact Number"
    },
    {
        "type": "QUESTION_SET",
        "repeatable": true,
        "subQuestions": [
            {
                "label": "What service are you looking for?",
                "type": "DROP_DOWN",
                "options": [
                    {
                        "text": "Hair Cut - Men"
                    },
                    {
                        "text": "Hair Cut - Women"
                    },
                    {
                        "text": "Hair Cut - Baby"
                    }
                ]
            },
            {
                "label": "How many people?",
                "type": "NUMBER"
            }
        ]
    }
]

Response (Also what you get when you GET questions/<queue-id>):

[
    {
        "questionId": "<UUID>",
        "type": "SHORT_TEXT",
        "label": "What is your name?"
    },
    {
        "questionId": "<UUID>",
        "type": "MOBILE_NUMBER",
        "label": "Contact Nummber"
    },
    {
        "questionId": "<UUID>",
        "type": "QUESTION_SET",
        "repeatable": true,
        "subQuestions": [
            {
                "questionId": "<UUID>",
                "label": "What service are you looking for?",
                "type": "DROP_DOWN",
                "options": [
                    {
                        "id": "1",
                        "text": "Hair Cut - Men"
                    },
                    {
                        "id": "2",
                        "text": "Hair Cut - Women"
                    },
                    {
                        "id": "3",
                        "text": "Hair Cut - Baby"
                    }
                ]
            },
            {
                "questionId": "<UUID>",
                "label": "How many people?",
                "type": "NUMBER"
            }
        ]
    }
]

[daltonfury42]

A sample joiner answer:

Name: Philip Mathew
Hair Cut Women: Quantity: 4
Hair Cut Men: Quantity: 2

Json:

[
    {
        "questionId": "<UUID>",
        "type": "SHORT_TEXT",
        "label": "What is your name?",
        "response": "Philip Mathew"
    },
    {
        "questionId": "<UUID>",
        "type": "MOBILE_NUMBER",
        "label": "Contact Nummber",
        "response": "+919488391128"
    },
    {
        "questionId": "<UUID>",
        "type": "QUESTION_SET",
        "repeatable": true,
        "subQuestions": [
            {
                "questionId": "<UUID>",
                "label": "What service are you looking for?",
                "type": "DROP_DOWN",
                "options": [
                    {
                        "id": "1",
                        "text": "Hair Cut - Men"
                    },
                    {
                        "id": "2",
                        "text": "Hair Cut - Women"
                    },
                    {
                        "id": "3",
                        "text": "Hair Cut - Baby"
                    }
                ],
                "response": "1"
            },
            {
                "questionId": "<UUID>",
                "label": "How many people?",
                "type": "NUMBER",
                "response": "2"
            }
        ]
    },
    {
        "questionId": "<UUID>",
        "type": "QUESTION_SET",
        "repeatable": true,
        "subQuestions": [
            {
                "questionId": "<UUID>",
                "label": "What service are you looking for?",
                "type": "DROP_DOWN",
                "options": [
                    {
                        "id": "1",
                        "text": "Hair Cut - Men"
                    },
                    {
                        "id": "2",
                        "text": "Hair Cut - Women"
                    },
                    {
                        "id": "3",
                        "text": "Hair Cut - Baby"
                    }
                ],
                "response": "2"
            },
            {
                "questionId": "<UUID>",
                "label": "How many people?",
                "type": "NUMBER",
                "response": "4"
            }
        ]
    }
]

[daltonfury42]

How other users might take it:

A Restaurant Owner Could Feed the Menu as a question:

Name: Philip Mathew
Phone:  +919455783397
Group Size: 6
[Item: Tea, 4]
[Item: Lime Juice, 2]
[Item: Fried Rice, 3]
[Item: Butter Naan, 6]

A printshop:

Name: Philip Mathew
Phone:  +919455783397
[file: abc.pdf, size: A4, type: "Black n white", Copies: 4]
[file: efg.jpeg, size: A5, type: "Color, Glossy Finish". Copies: 2]

[maaverik]

This looks good. Just one thing I noticed is the first object in the QUESTION_SET subquestions list doesn't have an id attribute. Is this intentional? It would break the structure, right?

"subQuestions": [ { "label": "What service are you looking for?", "type": "DROP_DOWN", "options": [ { "id": "1", "text": "Hair Cut - Men" }, { "id": "2", "text": "Hair Cut - Women" }, { "id": "3", "text": "Hair Cut - Baby" } ] }, { "id": "<UUID>", "label": "How many people?", "type": "NUMBER" } ]

And in the same object, there are ids in each of the dropdown options which we could maybe give a different name to avoid confusion.

[daltonfury42]

I've updated it. Yeah, I missed an id there.

yes, we should give different names. I'm thinking questionId for the UUID stuff, and normal id for dropdown like stuff

[maaverik]

Yeah, that'll work.

[daltonfury42]

I've updated the above posts with a questionId field.

[pavan541cs]

@daltonfury42 The approach looks good

[akashkbaburajan]

Made a basic design of Queue settings field where we can incorporate this functionality. https://xd.adobe.com/view/99a3a298-8f0c-468a-6cf3-614e96f6f9f5-ef92/

[pavan541cs]

@daltonfury42 Can i pick this story and start the implementation. I think design is done right ?

[daltonfury42]

Please. I was thinking about this over the weekend. These are my thoughts:

customQuestions will be a PostgreSQL JSON datatype. https://www.postgresql.org/docs/9.4/datatype-json.html

1. Let name and contactNumber still stay as separate fields. We would just have to add this extra field customQuestions to the queue entity.
2. Lets then expose a POST endpoint to set the customQuestions for a queue. Only the owner of the queue should be allowed to set it
3. QueueStatusResponse should then also return the customQuestions to anyone who calls. (This is an unauthenticated / public endpoint)
4. Optionally, if we can validate the json, it would be really nice. We should define the schema in some format like JsonSchema or OpenAPI definition.
5. Follow Up Task: We want to allow users to upload files using the custom form. One of the custom question we need to support would be a file type. How do we submit files as part of a form. Can you think of the best way to do that, for backend support.
6. Follow up Task 2: Save the files uploaded to S3 (on prod and dev profile, local profile should save to disk)

Then remaining task would be on the frontend side... We would have to define an interface that lets the users define and also another that lets users fill the form.

Just to give you context, we want this queue management system to be a very generic product that anyone can use. So, a hospital might ask patient hospital related question (which department, patient age etc) , a restaurant might ask number of people, smoking/non-smoking etc. I was talking to a photocopy shop owner recently. He was pretty excited to use this to get people to submit print orders to the queue. Most people currently crowd at the shop front desk and documents are send to an email. So if there is a fair queue where you can submit orders, they can process more people faster. Remember the crowds at photocopy shops back in college before exams. :)

Task 5 is for photocopy shop usecase, for people to upload files to be printed

[daltonfury42]

If the above looks fine to you, I'd say, split this into multiple PRs. Maybe c(1,2,3) can be one PR and 4, 5, 6 can be individual/separate PRs.

[thehamzarocks]

We should also give creators the ability to modify forms for existing queues. As if there are long-lived queues, the requirements might change over time. We need to version these forms so that old answers are preserved.

Another option for creators who recreate queues daily - it would be helpful for them to have their existing forms saved for reuse with their new queues.

[daltonfury42]

Yes, these two are super critical. @thehamzarocks After our versioning discussion, now I think we don't need to version, as the question is also included along with the answer. So these two would be easy to implement, I believe.

[pavan541cs]

@daltonfury42 What should be the new endpoint ? POST /queue/{queueId}/customQuestions ?

[daltonfury42]

@thehamzarocks I'd favour /queue/{queueId}/questions more. or /queue/{queueId}/custom-questions. I don't think we should name url paths in java variable style

[pavan541cs]

@daltonfury42 What database are we using in PROD ?

[daltonfury42]

PostgreSQL

[pavan541cs]

H2 doesnt support JSONB, So I am planning to change from H2 to postgres on local too @daltonfury42

[daltonfury42]

I see. I wanted locally running backends to connect to a H2 database so that it will ease development. You don't have to manage schema changes, and also a newcomer doesn't have to worry about setting up a local DB.

What do you recommend? Local postgres or storing the data as string?

cc: @thehamzarocks

[thehamzarocks]

Hmm with H2, it’s very easy for new contributors to get set up. Also, unless we plan on doing operations on the json from within sql, I think it’s fine to store it as a string.

As far as I can see, we’ll only be doing get/post operations on the forms from UI. And if we rely too much on Postgres JSON functionality, we’ll be tied to it forever.

But if you see the need, then we can consider it.

[daltonfury42]

@pavan541cs let's store the json as string in DB, unless you think we should use jsonb

[pavan541cs]

{
    "questions": [
    {
        "type": "SHORT_TEXT",
        "label": "What is your name?"
    },
    {
        "type": "MOBILE_NUMBER",
        "label": "Contact Number"
    },
    {
        "type": "QUESTION_SET",
        "repeatable": true,
        "subQuestions": [
            {
                "label": "What service are you looking for?",
                "type": "DROP_DOWN",
                "options": [
                    {
                        "text": "Hair Cut - Men"
                    },
                    {
                        "text": "Hair Cut - Women"
                    },
                    {
                        "text": "Hair Cut - Baby"
                    }
                ]
            },
            {
                "label": "How many people?",
                "type": "NUMBER"
            }
        ]
    }
] 
}

[pavan541cs]

@daltonfury42 Just a small change, I changed the input payload by adding questions at the parent level. Can you confirm if it is fine ?