diff --git a/book/80-examples/000-example-designs.ipynb b/book/80-examples/000-example-designs.ipynb index 02c7bfa..1a93d46 100644 --- a/book/80-examples/000-example-designs.ipynb +++ b/book/80-examples/000-example-designs.ipynb @@ -8,105 +8,7 @@ { "cell_type": "markdown", "metadata": {}, - "source": [ - "## Overview of Examples\n", - "\n", - "This section presents a variety of schema examples that demonstrate different aspects of DataJoint design principles. Each example showcases specific patterns, relationships, and use cases ranging from classic database tutorials to real-world scientific applications.\n", - "\n", - "### Classic Database Examples\n", - "\n", - "These examples are adapted from well-known database tutorials to show how traditional relational designs map to DataJoint's entity-workflow model.\n", - "\n", - "#### [Classic Sales Database](010-classic-sales.ipynb)\n", - "The classic MySQL tutorial sales database featuring customers, orders, products, and employees. This example demonstrates:\n", - "- Traditional business database design\n", - "- Many-to-many relationships through order details\n", - "- Employee management hierarchies\n", - "- Product lines and order fulfillment\n", - "\n", - "#### [University Database](015-university.ipynb)\n", - "A university course enrollment system with students, courses, instructors, and enrollments. This example illustrates:\n", - "- Academic record keeping\n", - "- Course registration workflows\n", - "- Student-course many-to-many relationships\n", - "- Grade tracking and prerequisites\n", - "\n", - "#### [Nations Database](030-nations.ipynb)\n", - "A geographical database with countries, regions, and continents from the MariaDB tutorial. This example shows:\n", - "- Hierarchical geographical relationships\n", - "- Lookup tables for standardized data\n", - "- Regional groupings and classifications\n", - "\n", - "### Conceptual Examples\n", - "\n", - "These examples focus on specific design patterns and conceptual challenges in database modeling.\n", - "\n", - "#### [Hotel Reservation System](040-hotel.ipynb)\n", - "A complete hotel management system with rooms, reservations, check-ins, and check-outs. This design exercise demonstrates:\n", - "- Time-series data (room availability and pricing)\n", - "- Business rule enforcement (no double-booking, check-in requires reservation)\n", - "- Guest management and payment processing\n", - "- Temporal queries (who stayed when, room history)\n", - "\n", - "#### [Social Generations](044-generations.ipynb)\n", - "Defines social generations as birth cohorts sharing cultural and historical experiences. This example explores:\n", - "- Lookup tables for reference data\n", - "- Birth year ranges and generational characteristics\n", - "- Cultural archetypes and life principles\n", - "- Using DataJoint to model sociological concepts\n", - "\n", - "#### [Languages and Proficiency](050-languages.ipynb)\n", - "A many-to-many relationship design tracking people and their language skills using international standards. This comprehensive example demonstrates:\n", - "- **ISO 639-1 language codes**: Standardized two-letter codes for major world languages\n", - "- **CEFR proficiency levels**: Common European Framework of Reference for Languages (A1-C2)\n", - "- **Association tables**: `Proficiency` as a workflow step linking people, languages, and levels\n", - "- **Lookup tables**: Both `Language` and `CEFRLevel` with predefined, standardized content\n", - "- **International standards**: Using established frameworks ensures consistency and compatibility\n", - "- **Sample queries**: Finding multilingual individuals, proficient speakers, and language statistics\n", - "\n", - "### Organizational Examples\n", - "\n", - "These examples model organizational structures and hierarchies.\n", - "\n", - "#### [Management Hierarchy](060-management.ipynb)\n", - "Demonstrates self-referencing tables and organizational relationships. This example shows:\n", - "- Employees managing other employees (recursive relationships)\n", - "- Department structure with chairs and budgets\n", - "- Organizational hierarchies and reporting chains\n", - "- One-to-one relationships (department chairs)\n", - "- Normalized department data (codes, names, budgets, locations)\n", - "\n", - "### Scientific Examples\n", - "\n", - "Real-world scientific applications demonstrating DataJoint in research contexts.\n", - "\n", - "#### [Allen Common Coordinate Framework](065-allen-ccf.ipynb)\n", - "The Allen Brain Atlas coordinate system for mouse brain anatomy. This example illustrates:\n", - "- 3D spatial coordinates and brain structure atlases\n", - "- Hierarchical anatomical relationships\n", - "- Integration with neuroimaging data\n", - "- Real scientific data pipeline from the DataJoint Elements project\n", - "- Published standards in neuroscience research\n", - "\n", - "#### [Julia Sets](070-julia.ipynb)\n", - "Computational visualization of Julia sets from fractal mathematics. This example demonstrates:\n", - "- Computed tables for mathematical functions\n", - "- Parameter spaces and computational workflows\n", - "- Image generation and visualization\n", - "- Using DataJoint for reproducible computational art\n", - "- Automatic computation with `populate()`\n", - "\n", - "---\n", - "\n", - "Each example is self-contained with:\n", - "- Complete schema definitions\n", - "- Sample data population\n", - "- Example queries demonstrating key concepts\n", - "- Visual diagrams showing table relationships\n", - "- Code that you can run and modify\n", - "\n", - "Browse through these examples to see different design patterns in action and learn how to apply DataJoint principles to your own domains.\n" - ] + "source": "## Overview of Examples\n\nThis section presents a variety of schema examples that demonstrate different aspects of DataJoint design principles. Each example showcases specific patterns, relationships, and use cases ranging from classic database tutorials to real-world scientific applications.\n\n### Classic Database Examples\n\nThese examples are adapted from well-known database tutorials to show how traditional relational designs map to DataJoint's entity-workflow model.\n\n#### [Classic Sales Database](010-classic-sales.ipynb)\nThe classic MySQL tutorial sales database featuring customers, orders, products, and employees. This example demonstrates:\n- Traditional business database design\n- Many-to-many relationships through order details\n- Employee management hierarchies\n- Product lines and order fulfillment\n\n#### [University Database](015-university.ipynb)\nA university course enrollment system with students, courses, instructors, and enrollments. This example illustrates:\n- Academic record keeping\n- Course registration workflows\n- Student-course many-to-many relationships\n- Grade tracking and prerequisites\n\n#### [Nations Database](030-nations.ipynb)\nA geographical database with countries, regions, and continents from the MariaDB tutorial. This example shows:\n- Hierarchical geographical relationships\n- Lookup tables for standardized data\n- Regional groupings and classifications\n\n### Conceptual Examples\n\nThese examples focus on specific design patterns and conceptual challenges in database modeling.\n\n#### [Hotel Reservation System](040-hotel.ipynb)\nA complete hotel management system with rooms, reservations, check-ins, and check-outs. This design exercise demonstrates:\n- Time-series data (room availability and pricing)\n- Business rule enforcement (no double-booking, check-in requires reservation)\n- Guest management and payment processing\n- Temporal queries (who stayed when, room history)\n\n#### [Social Generations](044-generations.ipynb)\nDefines social generations as birth cohorts sharing cultural and historical experiences. This example explores:\n- Lookup tables for reference data\n- Birth year ranges and generational characteristics\n- Cultural archetypes and life principles\n- Using DataJoint to model sociological concepts\n\n#### [Languages and Proficiency](050-languages.ipynb)\nA many-to-many relationship design tracking people and their language skills using international standards. This comprehensive example demonstrates:\n- **ISO 639-1 language codes**: Standardized two-letter codes for major world languages\n- **CEFR proficiency levels**: Common European Framework of Reference for Languages (A1-C2)\n- **Association tables**: `Proficiency` as a workflow step linking people, languages, and levels\n- **Lookup tables**: Both `Language` and `CEFRLevel` with predefined, standardized content\n- **International standards**: Using established frameworks ensures consistency and compatibility\n- **Sample queries**: Finding multilingual individuals, proficient speakers, and language statistics\n\n### Organizational Examples\n\nThese examples model organizational structures and hierarchies.\n\n#### [Management Hierarchy](060-management.ipynb)\nDemonstrates self-referencing tables and organizational relationships. This example shows:\n- Employees managing other employees (recursive relationships)\n- Department structure with chairs and budgets\n- Organizational hierarchies and reporting chains\n- One-to-one relationships (department chairs)\n- Normalized department data (codes, names, budgets, locations)\n\n### Scientific Examples\n\nReal-world scientific applications demonstrating DataJoint in research contexts.\n\n#### [Allen Common Coordinate Framework](065-allen-ccf.ipynb)\nThe Allen Brain Atlas coordinate system for mouse brain anatomy. This example illustrates:\n- 3D spatial coordinates and brain structure atlases\n- Hierarchical anatomical relationships\n- Integration with neuroimaging data\n- Real scientific data pipeline from the DataJoint Elements project\n- Published standards in neuroscience research\n\n#### [Julia Fractals](070-fractals.ipynb)\nComputational visualization of Julia sets from fractal mathematics. This example demonstrates:\n- Computed tables for mathematical functions\n- Parameter spaces and computational workflows\n- Image generation and visualization\n- Using DataJoint for reproducible computational art\n- Automatic computation with `populate()`\n\n#### [Blob Detection Pipeline](075-blob-detection.ipynb)\nAn image analysis pipeline that detects bright blobs in images. This example demonstrates:\n- Master-part table relationships for computational workflows\n- Lookup tables for algorithm parameters\n- Computed tables with automatic population\n- Visualization of detection results\n- Atomic transactions for workflow integrity\n\n---\n\nEach example is self-contained with:\n- Complete schema definitions\n- Sample data population\n- Example queries demonstrating key concepts\n- Visual diagrams showing table relationships\n- Code that you can run and modify\n\nBrowse through these examples to see different design patterns in action and learn how to apply DataJoint principles to your own domains." } ], "metadata": { diff --git a/book/80-examples/010-classic-sales.ipynb b/book/80-examples/010-classic-sales.ipynb index 885b8b7..ae122d2 100644 --- a/book/80-examples/010-classic-sales.ipynb +++ b/book/80-examples/010-classic-sales.ipynb @@ -31,11 +31,6 @@ "Because we run against the canonical SQL definitions, you can compare the DataJoint diagrams directly with the diagrams that appear in business-database textbooks. It also illustrates how DataJoint fits into an existing workflow—you can adopt legacy schemas while gaining the benefits of workflow execution, cascading deletes, and DAG-based querying.\n" ] }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [] - }, { "cell_type": "code", "execution_count": 6, @@ -587,11 +582,6 @@ "\n", "From here you can explore the database using either SQL (`%sql`) or DataJoint operators. Because the schema follows the relational workflow model, joins such as `Order * Payment` or `Product * Order.Details` naturally trace the same business processes that the foreign keys encode. Try recreating some of the analytical queries from the MySQL tutorial using DataJoint’s restriction (`&`), projection (`.proj()`), and aggregation (`.aggr()`) operators to see how workflow-aware queries feel in a classic OLTP dataset.\n" ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [] } ], "metadata": { diff --git a/book/80-examples/016-university-queries.ipynb b/book/80-examples/016-university-queries.ipynb index d805f89..cdf4f38 100644 --- a/book/80-examples/016-university-queries.ipynb +++ b/book/80-examples/016-university-queries.ipynb @@ -1808,11 +1808,6 @@ "The restriction operators & and - can use other queries as conditions." ] }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [] - }, { "cell_type": "code", "execution_count": 16, @@ -2521,11 +2516,6 @@ "# Joins" ] }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [] - }, { "cell_type": "code", "execution_count": 22, @@ -3528,4 +3518,4 @@ }, "nbformat": 4, "nbformat_minor": 2 -} +} \ No newline at end of file diff --git a/book/80-examples/030-nations.ipynb b/book/80-examples/030-nations.ipynb index 8dd1e19..93117b9 100644 --- a/book/80-examples/030-nations.ipynb +++ b/book/80-examples/030-nations.ipynb @@ -10,33 +10,14 @@ { "cell_type": "markdown", "metadata": {}, - "source": [ - "The nation database is reproduced here from the MySQL tutorial under fair use principles for instructional purposes.\n", - "\n", - "The original is found at\n", - "https://www.mariadbtutorial.com/getting-started/mariadb-sample-database/" - ] + "source": "## Background\n\nThe Nations database is reproduced here from the MariaDB tutorial under fair use principles for instructional purposes.\n\nThe original is found at https://www.mariadbtutorial.com/getting-started/mariadb-sample-database/\n\nThis example demonstrates how DataJoint can work with existing SQL schemas. We:\n\n- **Load the canonical schema** exactly as published by the MariaDB team so you can compare the DataJoint rendition with the original SQL definitions.\n- **Use `spawn_missing_classes()`** to automatically create DataJoint table classes from an existing database schema.\n- **Visualize the structure** using DataJoint's diagram tool to understand the relationships between tables.\n\nThe Nations database contains geographical data including:\n- **Continents**: Major continental regions\n- **Regions**: Sub-regions within continents\n- **Countries**: Individual nations with their regional assignments\n- **Country Statistics**: Population and GDP data over time\n- **Languages**: Languages spoken in each country\n\n## Loading the Dataset" }, { "cell_type": "code", - "execution_count": 1, + "execution_count": null, "metadata": {}, "outputs": [], - "source": [ - "import pymysql\n", - "import os\n", - "pymysql.install_as_MySQLdb()\n", - "\n", - "connection_string = \"mysql://{user}:{password}@{host}\".format(\n", - " user=os.environ['DJ_USER'],\n", - " host=os.environ['DJ_HOST'],\n", - " password=os.environ['DJ_PASS']\n", - ")\n", - "\n", - "%load_ext sql\n", - "%sql $connection_string\n", - "%config SqlMagic.style = '_DEPRECATED_DEFAULT'" - ] + "source": "%load_ext sql\n%sql mysql+pymysql://dev:devpass@db" }, { "cell_type": "code", @@ -252,152 +233,22 @@ "SHOW TABLES in nation;" ] }, + { + "cell_type": "markdown", + "source": "## DataJoint Schema Diagram\n\nNow we use DataJoint to introspect the existing schema and generate Python table classes. The `spawn_missing_classes()` method automatically creates DataJoint table classes that mirror the existing database tables.", + "metadata": {} + }, { "cell_type": "code", - "execution_count": 4, + "execution_count": null, "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "[2025-01-15 01:37:13,084][INFO]: Connecting root@localhost:3306\n", - "[2025-01-15 01:37:13,126][INFO]: Connected root@localhost:3306\n" - ] - }, - { - "data": { - "image/svg+xml": [ - "\n", - "\n", - "\n", - "\n", - "\n", - "Continents\n", - "\n", - "\n", - "Continents\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "Regions\n", - "\n", - "\n", - "Regions\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "Continents->Regions\n", - "\n", - "\n", - "\n", - "\n", - "Countries\n", - "\n", - "\n", - "Countries\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "CountryLanguages\n", - "\n", - "\n", - "CountryLanguages\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "Countries->CountryLanguages\n", - "\n", - "\n", - "\n", - "\n", - "CountryStats\n", - "\n", - "\n", - "CountryStats\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "Countries->CountryStats\n", - "\n", - "\n", - "\n", - "\n", - "Guests\n", - "\n", - "\n", - "Guests\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "Languages\n", - "\n", - "\n", - "Languages\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "Languages->CountryLanguages\n", - "\n", - "\n", - "\n", - "\n", - "RegionAreas\n", - "\n", - "\n", - "RegionAreas\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "Regions->Countries\n", - "\n", - "\n", - "\n", - "\n", - "Vips\n", - "\n", - "\n", - "Vips\n", - "\n", - "\n", - "\n", - "\n", - "" - ], - "text/plain": [ - "" - ] - }, - "execution_count": 4, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "# Plot diagram in DataJoint\n", - "\n", - "import datajoint as dj\n", - "schema = dj.Schema('nation')\n", - "schema.spawn_missing_classes()\n", - "\n", - "dj.Diagram(schema)" - ] + "outputs": [], + "source": "import datajoint as dj\nschema = dj.Schema('nation')\nschema.spawn_missing_classes()\n\ndj.Diagram(schema)" + }, + { + "cell_type": "markdown", + "source": "## Sample Query\n\nHere we display the `CountryStats` table showing population and GDP data over time:", + "metadata": {} }, { "cell_type": "code", @@ -544,13 +395,6 @@ "source": [ "CountryStats()" ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [] } ], "metadata": { @@ -574,4 +418,4 @@ }, "nbformat": 4, "nbformat_minor": 2 -} +} \ No newline at end of file diff --git a/book/80-examples/040-hotel.ipynb b/book/80-examples/040-hotel.ipynb index 2547468..3c47903 100644 --- a/book/80-examples/040-hotel.ipynb +++ b/book/80-examples/040-hotel.ipynb @@ -4,42 +4,13 @@ "cell_type": "markdown", "id": "6e26075b", "metadata": {}, - "source": [ - "# Hotel\n", - "\n", - "## Assignment\n", - "\n", - "Design, populate, and query a database for a hotel reservation system with the following business rules:\n", - "\n", - "1. The hotel has a number of rooms of two types: Deluxe and Suite\n", - "2. For every night, some rooms are made available for reservation for a specific price.\n", - "3. A guest can make a reservation for an avavilable room for one night. The reservation must include credit card payment info. At most one reservation can be made per night per room.\n", - "4. A guest can check into a room that has been reserved. An attempt to check in without a reservation will generate an error.\n", - "5. A guest can check out only after checking in. An attempt to check out multiple times or check out without checking in will generate an error.\n", - "\n", - "Your Python code should provide the following:\n", - "\n", - "1. A section to create the tables. The design must be in 3rd normal form following the conventions discussed in class and enforcing the business rules above.\n", - "\n", - "2. Provide code to populate rooms and room availability with prices.\n", - "\n", - "3. The function `reserve_room(room, date, guest_name, credit_card)` to make a reservation. A script that populates at least 300 reservations (e.g. use `faker`)\n", - "\n", - "4. The functions `checkin(room, date)` and `checkout(room, date)` to check guests in and out. Write a script that invokes `checkin` and `checkout` for a buncha guests. Demonstrate that that the functions enforces the rules of the business.\n", - "\n", - "5. Write a query to list all guests who have stayed in a given room in 2021.\n", - "\n", - "6. Write a query to list all dates on which a specific guest stayed at the hotel.\n", - "\n" - ] + "source": "# Hotel Reservation System\n\n## Background\n\nThis example demonstrates a complete hotel management system with rooms, reservations, check-ins, and check-outs. It showcases how DataJoint's referential integrity enforces business rules automatically through the schema design.\n\n### Business Rules\n\nThe hotel system enforces the following business rules:\n\n1. The hotel has rooms of two types: **Deluxe** and **Suite**\n2. For every night, rooms are made available for reservation at a specific price\n3. A guest can make a reservation for an available room for one night, including credit card payment info. At most one reservation can be made per night per room.\n4. A guest can check into a room only after making a reservation. An attempt to check in without a reservation will generate an error.\n5. A guest can check out only after checking in. Duplicate check-outs or checking out without checking in will generate an error.\n\n### Design Concepts Demonstrated\n\n- **Time-series data**: Room availability and pricing varies by date\n- **Business rule enforcement**: Database constraints prevent double-booking and enforce check-in/check-out sequence\n- **Guest management**: Guest records are created on-demand during reservation\n- **Workflow dependencies**: `CheckIn` depends on `Reservation`, `CheckOut` depends on `CheckIn`\n- **Error handling**: Custom exceptions provide clear feedback when rules are violated" }, { "cell_type": "markdown", "id": "33e46762", "metadata": {}, - "source": [ - "# Define the database" - ] + "source": "## Schema Definition" }, { "cell_type": "code", @@ -274,9 +245,7 @@ "cell_type": "markdown", "id": "3bab6ec9", "metadata": {}, - "source": [ - "# Populate Room Availability" - ] + "source": "## Populate Room Availability" }, { "cell_type": "code", @@ -335,9 +304,7 @@ "cell_type": "markdown", "id": "4deda107", "metadata": {}, - "source": [ - "# Functions " - ] + "source": "## Helper Functions\n\nThese functions encapsulate the business logic for reservations, check-ins, and check-outs. They use DataJoint's error handling to enforce business rules:" }, { "cell_type": "code", @@ -418,9 +385,7 @@ "cell_type": "markdown", "id": "60740817", "metadata": {}, - "source": [ - "# Operations" - ] + "source": "## Operations Demo\n\nNow let's simulate hotel operations: making reservations, checking guests in, and checking out:" }, { "cell_type": "code", @@ -1248,4 +1213,4 @@ }, "nbformat": 4, "nbformat_minor": 5 -} +} \ No newline at end of file diff --git a/book/80-examples/801-trippy-one.ipynb b/book/80-examples/801-trippy-one.ipynb deleted file mode 100644 index f167d42..0000000 --- a/book/80-examples/801-trippy-one.ipynb +++ /dev/null @@ -1,964 +0,0 @@ -{ - "cells": [ - { - "cell_type": "code", - "execution_count": 1, - "metadata": {}, - "outputs": [], - "source": [ - "import pymysql\n", - "import os\n", - "pymysql.install_as_MySQLdb()\n", - "\n", - "connection_string = \"mysql://{user}:{password}@{host}\".format(\n", - " user=os.environ['DJ_USER'],\n", - " host=os.environ['DJ_HOST'],\n", - " password=os.environ['DJ_PASS']\n", - ")\n", - "\n", - "%load_ext sql\n", - "%sql $connection_string\n", - "\n", - "# This patches a recent bug in MySQL:\n", - "# https://community.deeplearning.ai/t/error-while-querying-db-from-jupyter-notebook/716351\n", - "%config SqlMagic.style = '_DEPRECATED_DEFAULT'" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "%sql --file 20210-dump.sql" - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "metadata": {}, - "outputs": [], - "source": [ - "import datajoint as dj" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "%%sql\n", - "SHOW SCHEMAS" - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "[2025-01-15 16:59:39,082][INFO]: Connecting root@localhost:3306\n", - "[2025-01-15 16:59:39,119][INFO]: Connected root@localhost:3306\n" - ] - } - ], - "source": [ - "schema = dj.Schema('primitive_mvt_export')" - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "metadata": {}, - "outputs": [ - { - "data": { - "image/svg+xml": [ - "\n", - "\n", - "\n", - "\n", - "\n", - "Clip\n", - "\n", - "\n", - "Clip\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "Condition\n", - "\n", - "\n", - "Condition\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "Grate\n", - "\n", - "\n", - "Grate\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "Monet2\n", - "\n", - "\n", - "Monet2\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "Movie\n", - "\n", - "\n", - "Movie\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "MovieClip\n", - "\n", - "\n", - "MovieClip\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "Scan\n", - "\n", - "\n", - "Scan\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "Session\n", - "\n", - "\n", - "Session\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "Sync\n", - "\n", - "\n", - "Sync\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "Trial\n", - "\n", - "\n", - "Trial\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "Trippy\n", - "\n", - "\n", - "Trippy\n", - "\n", - "\n", - "\n", - "\n", - "" - ], - "text/plain": [ - "" - ] - }, - "execution_count": 5, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "schema.spawn_missing_classes()\n", - "dj.Diagram(schema)\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": {}, - "outputs": [ - { - "ename": "SyntaxError", - "evalue": "unmatched ')' (3172439320.py, line 1)", - "output_type": "error", - "traceback": [ - "\u001b[0;36m Cell \u001b[0;32mIn[7], line 1\u001b[0;36m\u001b[0m\n\u001b[0;31m Trial)()\u001b[0m\n\u001b[0m ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m unmatched ')'\n" - ] - } - ], - "source": [ - "Trial)()" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print(Monet2().size_on_disk)\n", - "print(MovieClip().size_on_disk)\n", - "print(Trippy().size_on_disk)\n", - "print(Clip().size_on_disk)\n", - "print(Movie().size_on_disk)\n", - "print(Condition().size_on_disk)\n", - "print(Scan().size_on_disk)\n", - "print(Trial().size_on_disk)\n", - "print(Sync().size_on_disk)\n" - ] - }, - { - "cell_type": "code", - "execution_count": 11, - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "\n", - " \n", - " \n", - " \n", - " \n", - "
\n", - " \n", - " \n", - " \n", - "\n", - "\n", - "
\n", - "

animal_id

\n", - " id number\n", - "
\n", - "

session

\n", - " session index for the mouse\n", - "
\n", - "

scan_idx

\n", - " number of TIFF stack file\n", - "
20505812
\n", - " \n", - "

Total: 1

\n", - " " - ], - "text/plain": [ - "*animal_id *session *scan_idx \n", - "+-----------+ +---------+ +----------+\n", - "20505 8 12 \n", - " (Total: 1)" - ] - }, - "execution_count": 11, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "dj.U('animal_id', 'session', 'scan_idx') & (Trial - Sync)" - ] - }, - { - "cell_type": "code", - "execution_count": 12, - "metadata": {}, - "outputs": [ - { - "data": { - "image/svg+xml": [ - "\n", - "\n", - "\n", - "\n", - "\n", - "Clip\n", - "\n", - "\n", - "Clip\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "Condition\n", - "\n", - "\n", - "Condition\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "Grate\n", - "\n", - "\n", - "Grate\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "Monet2\n", - "\n", - "\n", - "Monet2\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "Movie\n", - "\n", - "\n", - "Movie\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "MovieClip\n", - "\n", - "\n", - "MovieClip\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "Scan\n", - "\n", - "\n", - "Scan\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "Session\n", - "\n", - "\n", - "Session\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "Sync\n", - "\n", - "\n", - "Sync\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "Trial\n", - "\n", - "\n", - "Trial\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "Trippy\n", - "\n", - "\n", - "Trippy\n", - "\n", - "\n", - "\n", - "\n", - "" - ], - "text/plain": [ - "" - ] - }, - "execution_count": 12, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "dj.Diagram(schema)" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "\n", - " \n", - " \n", - " \n", - " clips from movies\n", - "
\n", - " \n", - " \n", - " \n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "
\n", - "

movie_name

\n", - " short movie title\n", - "
\n", - "

clip_number

\n", - " clip index\n", - "
\n", - "

file_name

\n", - " full file name\n", - "
\n", - "

clip

\n", - " \n", - "
\n", - "

parent_file_name

\n", - " \n", - "
\n", - "

duration

\n", - " duration in seconds\n", - "
bigrun30bigrun_0030.mov=BLOB=run125-lowPolyJungle-1-0.mp4None
bigrun176bigrun_0176.mov=BLOB=run257-mountainLake-1-0.mp4None
bigrun292bigrun_0292.mov=BLOB=run82-ancientTomb-1-0.mp4None
finalrun8finalrun_00008.mov=BLOB=americandiner_005-1-5-70-0_54.165.28.255.mp4None
finalrun95finalrun_00095.mov=BLOB=americandiner_079-1-5-70-0_184.73.123.94.mp4None
finalrun280finalrun_00280.mov=BLOB=ancienttomb_068-1-4-70-0_34.239.124.192.mp4None
finalrun355finalrun_00355.mov=BLOB=ancienttomb_105-1-4-70-0_54.89.40.102.mp4None
finalrun361finalrun_00361.mov=BLOB=ancienttomb_107-1-4-70-0_54.165.28.255.mp4None
finalrun691finalrun_00691.mov=BLOB=lowpolyjungle_015-1-9-70-0_54.175.147.171.mp4None
finalrun863finalrun_00863.mov=BLOB=lowpolyjungle_113-1-9-70-0_54.159.174.165.mp4None
finalrun893finalrun_00893.mov=BLOB=lowpolyjungle_125-1-9-70-0_54.175.147.171.mp4None
finalrun926finalrun_00926.mov=BLOB=medievalsewer_009-1-6-70-0_107.23.36.15.mp4None
\n", - "

...

\n", - "

Total: 54

\n", - " " - ], - "text/plain": [ - "*movie_name *clip_number file_name clip parent_file_na duration \n", - "+------------+ +------------+ +------------+ +--------+ +------------+ +----------+\n", - "bigrun 30 bigrun_0030.mo =BLOB= run125-lowPoly None \n", - "bigrun 176 bigrun_0176.mo =BLOB= run257-mountai None \n", - "bigrun 292 bigrun_0292.mo =BLOB= run82-ancientT None \n", - "finalrun 8 finalrun_00008 =BLOB= americandiner_ None \n", - "finalrun 95 finalrun_00095 =BLOB= americandiner_ None \n", - "finalrun 280 finalrun_00280 =BLOB= ancienttomb_06 None \n", - "finalrun 355 finalrun_00355 =BLOB= ancienttomb_10 None \n", - "finalrun 361 finalrun_00361 =BLOB= ancienttomb_10 None \n", - "finalrun 691 finalrun_00691 =BLOB= lowpolyjungle_ None \n", - "finalrun 863 finalrun_00863 =BLOB= lowpolyjungle_ None \n", - "finalrun 893 finalrun_00893 =BLOB= lowpolyjungle_ None \n", - "finalrun 926 finalrun_00926 =BLOB= medievalsewer_ None \n", - " ...\n", - " (Total: 54)" - ] - }, - "execution_count": 15, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "MovieClip()" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "dj.config['display.limit'] = 20" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Sync()" - ] - }, - { - "cell_type": "code", - "execution_count": 16, - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "\n", - " \n", - " \n", - " \n", - " movies used for generating clips and stills\n", - "
\n", - " \n", - " \n", - " \n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "
\n", - "

movie_name

\n", - " short movie title\n", - "
\n", - "

path

\n", - " \n", - "
\n", - "

movie_class

\n", - " \n", - "
\n", - "

original_file

\n", - " \n", - "
\n", - "

file_template

\n", - " filename template with full path\n", - "
\n", - "

file_duration

\n", - " (s) duration of each file (must be equal)\n", - "
\n", - "

codec

\n", - " \n", - "
\n", - "

movie_description

\n", - " full movie title\n", - "
\n", - "

frame_rate

\n", - " frames per second\n", - "
\n", - "

frame_width

\n", - " pixels\n", - "
\n", - "

frame_height

\n", - " pixels\n", - "
bigrunW:\\UnrealVideos\\bigrun1unrealrun*.mp4bigrun_%04u.mov70.0-c:v libx264 -vf format=gray -pix_fmt yuv420p -s 256:144 -preset slow -crf 10rendered movies at various environments30.0256144
finalrun/scratch05/UnrealVideos/rendered_moviesunreal*.mp4finalrun_%05u.mov60.0-ss 9.966666667 -c:v libx264 -vf format=gray -pix_fmt yuv420p -s 256:144 -preset slow -crf 10large set of rendered. Last 60\" without the last frame30.0256144
koyqatsiH:\\QATSI\\KOYAANISQATSIcinemakoyaanisqatsi_2.mp4koyqatsi_%03u.mov60.0-c:v libx264 -vf format=gray -pix_fmt yuv420p -preset slow -crf 10KOYAANISQATSI30.0256144
MadMax~/stimuli/movies/madmaxcinemamadmax.avimadmax_%03u.mov60.0-c:v libx264 -vf format=gray -pix_fmt yuv420p -preset slow -crf 10Mad Max: Fury Road (2015)30.0256144
matrixH:\\THE_MATRIXcinemamatrix.mp4matrix_%03u.mov60.0-c:v libx264 -vf format=gray -pix_fmt yuv420p -preset slow -crf 10Matrix30.0256144
matrixrlH:\\THE_MATRIX_RELOADEDcinemamatrix-reloaded_1.mp4matrixrl_%03u.mov60.0-c:v libx264 -vf format=gray -pix_fmt yuv420p -preset slow -crf 10Matrix reloaded30.0256144
matrixrvH:\\THE_MATRIX_REVOLUTIONScinemamatrx_rev.mp4matrixrv_%03u.mov60.0-c:v libx264 -vf format=gray -pix_fmt yuv420p -preset slow -crf 10Matrix revolutions30.0256144
naqatsiF:\\QATSI\\NAQOYQATSIcinemaNAQOYQATSI_2.mp4naqatsi_%03u.mov60.0-c:v libx264 -vf format=gray -pix_fmt yuv420p -preset slow -crf 10NAQOYQATSI30.0256144
poqatsiF:\\QATSI\\POWAQQATSIcinemaPOWAQQATSI_2.mp4poqatsi_%03u.mov60.0-c:v libx264 -vf format=gray -pix_fmt yuv420p -preset slow -crf 10POWAQQATSI30.0256144
sports1mW:\\MLDatasets\\sports1myoutube*.mp4sports1m_%06u.mov60.0-c:v libx264 -vf \"format=gray,crop=in_w*0.8:in_w*0.8*9/16,scale=256:144\" -pix_fmt yuv420p -preset slow -crf 10sports1m from youtube30.0256144
starwarsF:\\Starwarscinemavideo_3.mp4starwars_%03u.mov60.0-c:v libx264 -vf format=gray -pix_fmt yuv420p -preset slow -crf 10Starwars - The force awakens30.0256144
\n", - " \n", - "

Total: 11

\n", - " " - ], - "text/plain": [ - "*movie_name path movie_class original_file file_template file_duration codec movie_descript frame_rate frame_width frame_height \n", - "+------------+ +------------+ +------------+ +------------+ +------------+ +------------+ +------------+ +------------+ +------------+ +------------+ +------------+\n", - "bigrun W:\\UnrealVideo unreal run*.mp4 bigrun_%04u.mo 70.0 -c:v libx264 - rendered movie 30.0 256 144 \n", - "finalrun /scratch05/Unr unreal *.mp4 finalrun_%05u. 60.0 -ss 9.96666666 large set of r 30.0 256 144 \n", - "koyqatsi H:\\QATSI\\KOYAA cinema koyaanisqatsi_ koyqatsi_%03u. 60.0 -c:v libx264 - KOYAANISQATSI 30.0 256 144 \n", - "MadMax ~/stimuli/movi cinema madmax.avi madmax_%03u.mo 60.0 -c:v libx264 - Mad Max: Fury 30.0 256 144 \n", - "matrix H:\\THE_MATRIX cinema matrix.mp4 matrix_%03u.mo 60.0 -c:v libx264 - Matrix 30.0 256 144 \n", - "matrixrl H:\\THE_MATRIX_ cinema matrix-reloade matrixrl_%03u. 60.0 -c:v libx264 - Matrix reloade 30.0 256 144 \n", - "matrixrv H:\\THE_MATRIX_ cinema matrx_rev.mp4 matrixrv_%03u. 60.0 -c:v libx264 - Matrix revolut 30.0 256 144 \n", - "naqatsi F:\\QATSI\\NAQOY cinema NAQOYQATSI_2.m naqatsi_%03u.m 60.0 -c:v libx264 - NAQOYQATSI 30.0 256 144 \n", - "poqatsi F:\\QATSI\\POWAQ cinema POWAQQATSI_2.m poqatsi_%03u.m 60.0 -c:v libx264 - POWAQQATSI 30.0 256 144 \n", - "sports1m W:\\MLDatasets\\ youtube *.mp4 sports1m_%06u. 60.0 -c:v libx264 - sports1m from 30.0 256 144 \n", - "starwars F:\\Starwars cinema video_3.mp4 starwars_%03u. 60.0 -c:v libx264 - Starwars - The 30.0 256 144 \n", - " (Total: 11)" - ] - }, - "execution_count": 16, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "Movie()" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "Sync()" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "%%sql\n", - "SHOW databases" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "Trial - Sync" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "Sync" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "Sync()" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [] - } - ], - "metadata": { - "kernelspec": { - "display_name": "base", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.11.11" - } - }, - "nbformat": 4, - "nbformat_minor": 2 -}