Noise in image to text data-set.

[codeorbit]

@prodicus Dataset which contain text from images is not cleaned (i.e containing lots of special character and numbers) and not normalized as well.
for e.g. sambhar vada , vada sambar, vada with sambhar all are same but they are different in the dataset.

[tasdikrahman]

@codeorbit That would be a problem.

But it would be a pain to manually clean the dataset with the cases like sambhar vada. Skim through the dataset once, there are some properly OCR'd menus in there.

Does nltk (or any other module) have something which normalizes the data the way we want?

[codeorbit]

@prodicus There is no such module for that in nltk. Everyone makes their own custom algorithm for normalization based on some pattern in the dataset.

[tasdikrahman]

@codeorbit So would your models not work with this data?

[codeorbit]

Not only mine, any model will not work with noisy data.

[tasdikrahman]

What is the plan now then? OCR wont give be giving us the data in the form you want. Burrp hosts text menus on their website. So we could probably use them

[codeorbit]

@prodicus Cool .. we can go for it 👍