feat: add configurable robots meta tag

[nsprenkle]

Enable config-based setting of <meta name="robots" content="{value}"> tag to allow Learning MFE to instruct search crawlers / indexers how to handle its content.

Related PRs:

• feat: allow setting robots content attribute through config frontend-build#611

See also:

• Robots meta tag reference
• AU-2285

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 89.25%. Comparing base (c20c767) to head (697f7a6).
Report is 1 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #1520   +/-   ##
=======================================
  Coverage   89.25%   89.25%           
=======================================
  Files         318      318           
  Lines        5563     5563           
  Branches     1379     1341   -38     
=======================================
  Hits         4965     4965           
  Misses        583      583           
  Partials       15       15           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

---

🚨 Try these New Features:

• Flaky Tests Detection - Detect and resolve failed and flaky tests
• JS Bundle Analysis - Avoid shipping oversized bundles