Spidering https://www.mit.edu/

1. Retrieving: https://www.mit.edu/ at 23:16:44.
Size of page: 48.89kb. Starting indexing at 23:16:45.
Indexed
Links found: 54. New links: 54
2. Retrieving: http://calendar.mit.edu/ at 23:17:08.
Size of page: 120.29kb. Starting indexing at 23:17:09.
Indexed
Links found: 167. New links: 165
3. Retrieving: http://careers.mit.edu/ at 23:18:08.
Relocation: http 301
Links found: 0. New links: 0
4. Retrieving: http://news.mit.edu/ at 23:18:13.
Size of page: 184.61kb. Starting indexing at 23:18:19.
Indexed
Links found: 117. New links: 106
5. Retrieving: http://news.mit.edu/2022/bringing-hope-transformation-congo-milain-fayulu-0524 at 23:19:01.
Size of page: 123.02kb. Starting indexing at 23:19:02.
Indexed
Links found: 129. New links: 45
6. Retrieving: http://news.mit.edu/2022/gudni-thorlacius-johannesson-iceland-0524 at 23:19:25.
Size of page: 127.92kb. Starting indexing at 23:19:31.
Indexed
Links found: 117. New links: 31
7. Retrieving: http://news.mit.edu/2022/mit-simmons-hall-interactive-canvas-0525 at 23:19:50.
Size of page: 115.65kb. Starting indexing at 23:19:56.
Indexed
Links found: 111. New links: 21
8. Retrieving: http://news.mit.edu/2022/ngozi-okonjo-iweala-commencement-2022 at 23:20:11.
Size of page: 149.50kb. Starting indexing at 23:20:17.
Indexed
Links found: 121. New links: 31
9. Retrieving: http://news.mit.edu/2022/ngozi-okonjo-iweala-commencement-2022-0217 at 23:20:35.
Size of page: 125.61kb. Starting indexing at 23:20:41.
Indexed
Links found: 111. New links: 16
10. Retrieving: http://news.mit.edu/2022/voice-for-change-jessica-chomik-morales-0524 at 23:20:54.
Size of page: 119.32kb. Starting indexing at 23:21:01.
Indexed
Links found: 113. New links: 25
11. Retrieving: http://socialmediahub.mit.edu/ at 23:21:18.
Size of page: 16.23kb. Starting indexing at 23:21:24.
Indexed
Links found: 30. New links: 25
12. Retrieving: http://whereis.mit.edu/ at 23:21:34.
Size of page: 7.66kb. Starting indexing at 23:21:40.
Indexed
Links found: 17. New links: 16
13. Retrieving: https://commencement.mit.edu/webcasts at 23:21:48.
Size of page: 59.88kb. Starting indexing at 23:21:50.
Indexed
Links found: 59. New links: 56
14. Retrieving: https://twitter.com/mit at 23:22:06.
Size of page: 98.61kb. Starting indexing at 23:22:09.
Indexed
Links found: 68. New links: 67
15. Retrieving: https://twitter.com/share?url=http://web.mit.edu/spotlight/congratulations-class-2022-0 at 23:22:25.
Size of page: 98.72kb. Starting indexing at 23:22:28.
Indexed
Links found: 68. New links: 48
16. Link https://www.facebook.com/MITnews: file checking forbidden by required/disallowed string rule
17. Retrieving: https://www.facebook.com/sharer/sharer.php?u=http://web.mit.edu/spotlight/congratulations-class-2022-0 at 23:22:40.
Size of page: 15.75kb. Starting indexing at 23:22:42.