Fix: robots.txt and site indexing

[cp-sumi-k]

Review checklist

Note: Make sure all points should be checked before submiting PR for review, otherwise PR will be cosidered invalid.

• Follow guidelines for creating branch name, adding commit messages or PR title
• Add MR description if applicable
• Follow guidelines for naming conventions everywhere (i.e files/folders, data structures)
• Reuse code (if the same code is written twice, make it common and reuse it at both places)
• Remove unused or commented code if not required
• Make MR self-approved - review MR as a reviewer and give it self-approval if everything is ok, if not make the required changes
• In progress MR should be marked as draft
• Make MR mark as ready before submitting it for review
• Add images, videos, or gifs of UI
• Add page speed score screenshot of UI (It should be > 85 for mobile and > 90 for desktops)

Design should be tested on:

• Inspector (all iPhones and android devices like samsung, motorola)
• Desktop (chrome, firefox, safari)
• Laptop / Macbook (chrome, firefox, safari)
• iPhone(safari)
• Available android devices (chrome, firefox)
• iPad(safari)
• Tablet (chrome, firefox)