WebBuilding

Organizations

Internet Archive

Summary

The Internet Archive is a nonprofit digital library in San Francisco, founded by Brewster Kahle in 1996 to preserve the public web and other digital material. Its Wayback Machine, opened to the public on October 24, 2001, lets anyone view earlier versions of web pages, and passed one trillion archived pages in October 2025. It also lends digitized books and holds audio, video and software collections.

Web Building Encyclopedia · Organizations · Last reviewed October 22, 2026 · 8 min read

What the Internet Archive is

The Internet Archive is a nonprofit digital library based in San Francisco. Brewster Kahle founded it in 1996 to collect and keep the public material of the internet, starting with pages of the World Wide Web.12

Its best-known service is the Wayback Machine, which stores copies of web pages as they looked on the day they were collected. Anyone can type in a web address and see earlier versions of that page.3

The Archive describes itself as a 501(c)(3) nonprofit, a charity under United States tax law, and says its mission is to give researchers, historians and scholars permanent access to historical collections in digital form.4 Besides web pages, it holds movies, audio recordings, books and other texts, and software.4

Its Form 990 for the year ending December 2025, the annual return a nonprofit files with the IRS, reported total revenue of $23,709,162 and total expenses of $27,766,880, according to ProPublica’s Nonprofit Explorer.5

In October 2025 the Wayback Machine passed one trillion archived web pages.6 At the celebration, Vint Cerf said that without this work “the 22nd century will have no clue what the 21st Century was all about.”6

Founding: Brewster Kahle and “Preserving the Internet”

Kahle studied artificial intelligence at MIT, where he earned a bachelor’s degree in 1982, and helped start the supercomputer maker Thinking Machines in 1983.1 In 1989 he created WAIS, which his biography describes as the internet’s first publishing system, and founded WAIS Inc., which was sold to America Online in 1995.1

In April 1996 Kahle co-founded Alexa Internet, a company that catalogued the web, which was sold to Amazon in 1999.1 The Archive’s own blog dates his founding of the Internet Archive to the same month, April 1996.2

The New York Times reported the project in June 1996. Kahle and five staff members worked from offices in the former Presidio army base in San Francisco, and they had already archived the text of the web using data donated by a web crawler company.7 The paper said Kahle was paying for the work with money from the sale of WAIS.7 The Archive’s anniversary timeline dates the first pages captured by its own crawlers to October 1996, among them MTV’s site.8

Kahle set out the idea in an essay, “Preserving the Internet,” submitted to Scientific American on November 4, 1996, and published in March 1997.2 It estimated the web at about 50 million pages, roughly 1.5 terabytes, and warned that “the average lifetime of a document is 75 days and then it is gone.” At the time the Archive held about 600 gigabytes.2

Early partners included the Smithsonian, which worked with the Archive on 1996 presidential campaign sites for a museum exhibit.7 On August 1, 2000, the Library of Congress and the Internet Archive set up the U.S. Election 2000 Web Archive.8

The Wayback Machine

The Archive collected pages for five years before it opened them to the public. It launched the Wayback Machine on October 24, 2001, at the University of California, Berkeley’s Bancroft Library.3 The launch press release counted more than 10 billion pages dating back to 1996, about 100 terabytes of data growing by about 10 terabytes a month. It listed the Library of Congress and the Smithsonian Institution among the Archive’s collaborators, and Alexa Internet among its benefactors.3

How pages are collected and stored

Pages are gathered by crawlers, programs that request a page, save it and follow its links to find more. The Archive’s own crawler, Heritrix, is described in its code repository as an “archival-quality web crawler” and is free software under the Apache License 2.0.9

Crawled material is stored in WARC files. The Library of Congress describes WARC as a revision of the Archive’s earlier ARC format, approved as the international standard ISO 28500 in May 2009.10 Each WARC record keeps the saved content along with details of the capture, such as the time of the harvest, the file type and the HTTP response code.10

How people use it

A visitor enters a domain or URL, and the Wayback Machine shows a calendar of the days it holds captures. The dots are colored by the response the crawler received: blue for a normal page, green for a redirect, orange for a client error such as a missing page, and red for a server error.11

Every archived address carries a 14-digit timestamp in the order year, month, day, hour, minute and second. There is a lag of 3 to 10 hours between a crawl and the moment it appears in the Wayback Machine.11

The Save Page Now feature lets anyone save a single page once. It does not add the address to future crawls and cannot save a whole site.11

Some pages do not archive well. The help center lists pages behind passwords, JavaScript that depends on the original server, server-side image maps and orphan pages that nothing links to, and notes that simple HTML is the easiest to archive.11

Programs can ask the Wayback Machine for the closest saved copy of a page through its availability API. The answer comes back in JSON, and an optional timestamp asks for the capture nearest a given date.12

Links from other services

From September 11, 2024, Google Search added a link to a site’s Wayback Machine page inside its “About this result” panel. The Archive said the link would not appear where a rights holder had opted out of archiving.13 With the Wikimedia Foundation, the Archive says it has found and fixed more than 28 million broken links on Wikipedia.6

The Archive announced in July 2025 that it expected to pass one trillion pages that October, and held a celebration at its San Francisco headquarters on October 22, 2025.146

Beyond web pages: books, media and software

The Archive’s timeline lists a run of projects built on the same storage and staff:8

  • Petabox, 2004. The Archive built its own server and storage system.
  • Archive-It, November 5, 2005. A web archiving service for other organizations.
  • Open Library, July 16, 2007. Aaron Swartz announced its launch.
  • TV News Archive, September 18, 2012. A collection of television news.
  • Historical Software Archive, October 24, 2013. A collection of historical software.

Open Library’s stated goal is “one web page for every book ever published.” As of October 2026 its About page lists more than 20 million catalog records and more than 3 million books to read or borrow free.15

On June 25, 2007, Kahle announced that the State of California had officially recognized the Internet Archive as a library. He noted that some federal funding requires state certification that an organization is a library.16

The Archive marked its 30th anniversary in 2026. In May it began a “Class of 1996” series that links to Wayback Machine snapshots of sites and services from the early web, mostly captured between 1996 and 1999.17

Robots.txt, exclusions and what gets saved

A robots.txt file sits at the top of a website and tells automated crawlers which parts of the site to leave alone. Heritrix is designed to respect robots.txt rules and META nofollow tags.9

In the past, a robots.txt block could also remove a domain’s earlier captures from view in the Wayback Machine, not just stop future crawls.18 Mark Graham, director of the Wayback Machine, wrote on April 17, 2017, that this caused trouble when a live site became a parked domain, because the parked domain’s robots.txt could remove the whole history of the site. He said complaints about such vanished sites arrived almost daily.18

In the same post he said the Archive had stopped honoring robots.txt on United States government and military sites a few months earlier, for both crawling and display, and was looking to do the same more broadly.18 He described the goal as “representing the web as it really was, and is, from a user’s perspective.”18

The current help pages still say a site may be missing because robots.txt blocked it or because its owner asked to be excluded.11 The documented route for removal is an email to [email protected]. Each request is reviewed, and the Archive makes no guarantee about the outcome.19

Archive-It crawls, run for partner organizations, identify themselves as archive.org_bot. Standard Archive-It crawls respect robots.txt and crawl delays by default, but an “Ignore Robots.txt” rule can be added to a site or host at the collection level.20

Lawsuits and the 2024 cyberattacks

Book lending: Hachette v. Internet Archive

The Archive lent scanned books under a model called controlled digital lending, which kept the number of digital loans at or below the number of print copies it owned.21 On March 24, 2020, during COVID-19 closures, it launched the National Emergency Library.8 That program lifted the one-to-one limit and let up to 10,000 people borrow a book at once, from March to June 2020.21

On June 1, 2020, four publishers, Hachette, HarperCollins, John Wiley and Sons, and Penguin Random House, sued in federal court in New York over 127 books.2221 On March 24, 2023, Judge John G. Koeltl granted summary judgment to the publishers and ruled that the Archive’s fair use defense “fails as a matter of law.”22

The Court of Appeals for the Second Circuit affirmed on September 4, 2024. It disagreed with the lower court on one point, finding the Archive’s use was not commercial, but held that the use was not transformative and that the fair use defense failed.21 On December 4, 2024, the Archive said it would not ask the Supreme Court to review the case and would keep honoring its agreement to remove books from lending when member publishers of the Association of American Publishers ask.23

Old records: the Great 78 Project case

On August 11, 2023, record companies including UMG Recordings, Capitol Records and Sony Music Entertainment sued the Archive in federal court in Northern California over the Great 78 Project, which, according to the complaint as summarized by the court, posted digitized 78 rpm records for free streaming and download.24 The amended complaint listed 4,142 recordings, and on May 15, 2024, Judge Maxine M. Chesney denied the defendants’ motions to dismiss.24 On September 15, 2025, the Archive wrote that the parties had told the court they had resolved all claims on confidential terms.25

October 2024 attacks

The Archive was hit by cyberattacks on October 9, 2024. Attackers disclosed archive.org email addresses and encrypted passwords to a transparency website and sent emails to users through a third-party help desk system.26 The Wayback Machine came back on October 13, Archive-It on October 17, and archive.org returned in read-only form on October 21.26

Timeline

  1. The Archive starts a "Class of 1996" series for its 30th anniversary17
  2. The Wayback Machine passes one trillion archived web pages6
  3. The Archive says the Great 78 Project lawsuit has been settled25
  4. The Archive says it will not seek Supreme Court review in Hachette23
  5. Cyberattacks take archive.org services offline26
  6. Google Search adds links to Wayback Machine pages13
  7. The Second Circuit rules against the Archive in Hachette21
  8. Record companies sue over the Great 78 Project24
  9. District court grants summary judgment to the publishers22
  10. Four publishers sue the Archive over book lending22
  11. The Archive explains why it is moving away from robots.txt rules18
  12. California recognizes the Internet Archive as a library16
  13. Archive-It web archiving service launches8
  14. The Wayback Machine opens to the public at UC Berkeley3
  15. Brewster Kahle founds the Internet Archive2

What it means for your website

You do not need an account to use the Wayback Machine, and it is often the only record of what an old site looked like. It is most useful for a small business in five situations.

  • See your own site’s history. Enter your domain at web.archive.org and pick a date on the calendar. Blue dots are the safest choices, because they mark normal page captures.11
  • List your old addresses before a move. The help center’s wildcard search, web.archive.org/*/yoursite.com/*, lists every archived address on a domain.11 That list is a good starting point for the 301 redirects in a website migration and for the redesign checklist.
  • Do not treat it as a backup. It saves what visitors see, not your database, forms or server code, and some JavaScript-heavy pages do not render.11 Keep real website backups, and use the Wayback Machine only to recover text and images you no longer have.
  • Check a used domain before you buy it. Old captures show what an expired domain name hosted before. Our guide to buying and selling websites covers the other checks.
  • Save one page as a dated record. Save Page Now stores a single page once, for example your terms or a price list on a given day. It does not schedule future crawls.11

If you want your site archived, make sure robots.txt and meta robots tags do not tell crawlers to stay away.11 If you want pages removed, robots.txt is not a dependable switch: past policy changes moved away from it, and Archive-It collections can be set to ignore it.1820 Instead, email [email protected] with the URLs, the time period you want excluded, the period you controlled the site (if applicable) and any other details that explain the request. The Archive reviews each request and does not promise a result.19 Copyright complaints go through its copyright policy.19

See also

References

  1. Brewster Kahle: About. Brewster Kahle, brewster.kahle.org. accessed October 22, 2026.
  2. Looking back on "Preserving the Internet" from 1996. Chris Freeland, Internet Archive Blogs. 2025-09-02. accessed October 22, 2026.
  3. Internet Archive Launches Wayback Machine (press release, forwarded to the liblicense-l list). liblicense-l archive, Center for Research Libraries. 2001-10-24. accessed October 22, 2026.
  4. About the Internet Archive. Internet Archive Blogs. accessed October 22, 2026.
  5. Internet Archive, Nonprofit Explorer (Form 990 data). ProPublica. accessed October 22, 2026.
  6. One Trillion Web Pages Archived: Internet Archive Celebrates a Civilization-Scale Milestone. Caralee Adams, Internet Archive Blogs. 2025-10-31. accessed October 22, 2026.
  7. Project Aims to Archive the Entire Internet (Laurie J. Flynn, The New York Times, June 1996, forwarded to the SERIALST list). SERIALST mailing list archive, Simplelists. 1996-06-25. accessed October 22, 2026.
  8. Internet Archive 25th anniversary timeline. Internet Archive. accessed October 22, 2026.
  9. Heritrix (heritrix3 repository README). Internet Archive, GitHub. accessed October 22, 2026.
  10. WARC, Web ARChive file format. Sustainability of Digital Formats, Library of Congress. accessed October 22, 2026.
  11. Using the Wayback Machine. Internet Archive Help Center. accessed October 22, 2026.
  12. Wayback Machine APIs: Wayback Availability JSON API. Internet Archive. 2013-09-24. accessed October 22, 2026.
  13. New Feature Alert: Access Archived Webpages Directly Through Google Search. Chris Freeland, Internet Archive Blogs. 2024-09-11. accessed October 22, 2026.
  14. Wayback Machine to Hit 'Once-in-a-Generation Milestone' this October: One Trillion Web Pages Archived. Chris Freeland, Internet Archive Blogs. 2025-07-01. accessed October 22, 2026.
  15. About Open Library. Open Library, Internet Archive. accessed October 22, 2026.
  16. Internet Archive officially a library. Brewster Kahle, Internet Archive Blogs. 2007-06-25. accessed October 22, 2026.
  17. Celebrating Thirty Years of the Internet Archive with the 'Class of 1996'. Chris Freeland, Internet Archive Blogs. 2026-05-07. accessed October 22, 2026.
  18. Robots.txt meant for search engines don't work well for web archives. Mark Graham, Internet Archive Blogs. 2017-04-17. accessed October 22, 2026.
  19. How do I request to remove something from archive.org?. Internet Archive Help Center. accessed October 22, 2026.
  20. Robots.txt exclusions and how they can impact your web archives. Archive-It Help Center, Internet Archive. accessed October 22, 2026.
  21. Hachette Book Group, Inc. v. Internet Archive, No. 23-1260, Opinion. U.S. Court of Appeals for the Second Circuit, via GovInfo. 2024-09-04. accessed October 22, 2026.
  22. Hachette Book Group, Inc. v. Internet Archive, No. 20-cv-4160 (JGK), Opinion and Order. U.S. District Court for the Southern District of New York (copy hosted by Frankfurt Kurnit Klein and Selz). 2023-03-24. accessed October 22, 2026.
  23. End of Hachette v. Internet Archive. Chris Freeland, Internet Archive Blogs. 2024-12-04. accessed October 22, 2026.
  24. UMG Recordings, Inc. v. Internet Archive, No. 23-cv-06522-MMC, Order Denying Defendants' Motions to Dismiss. U.S. District Court for the Northern District of California, via CourtListener RECAP. 2024-05-15. accessed October 22, 2026.
  25. An Update on the Great 78s Lawsuit. Chris Freeland, Internet Archive Blogs. 2025-09-15. accessed October 22, 2026.
  26. Internet Archive Services Update: 2024-10-21. Chris Freeland, Internet Archive Blogs. 2024-10-21. accessed October 22, 2026.

Web Building Encyclopedia. First published October 22, 2026. Last reviewed October 22, 2026. Every fact on this page is linked to a source we opened. Spotted an error? Write to [email protected].