It lives at yourdomain.com/robots.txt and consists of User-agent lines naming a crawler and Allow or Disallow lines for paths. A well behaved crawler reads it before anything else. It can also declare where the sitemap is.
It is not security, since anything can ignore it, but every search engine and AI crawler that matters respects it. That makes it powerful in both directions: one wrong line hides a site from Google, and one line for GPTBot removes it from ChatGPT's answers. It should be short, deliberate and checked after every platform change.