Skip to content
Snippets Groups Projects

Compare revisions

Changes are shown as if the source revision was being merged into the target revision. Learn more about comparing revisions.

Source

Select target project
No results found

Target

Select target project
  • sacs/cs-449-sds-public/project/cs449-template-m2-2022
  • hlanfran/cs449-template-m2-2022
2 results
Show changes
Commits on Source (54)
Showing
with 2590 additions and 19 deletions
.DS_Store
.bsp
**/*.swp
data/.DS_Store
data/ml-100k
data/ml-25m
project/project
project/target
src/main/scala/project/
src/main/scala/target/
target/
logs/
\documentclass{article}
\usepackage{hyperref}
\usepackage{algorithm}
\usepackage{algpseudocode}
\usepackage{ dsfont }
\usepackage{amsmath}
\usepackage{filemod}
\usepackage{ulem}
\usepackage{graphicx}
\usepackage{todonotes}
\input{Milestone-2-questions.sty}
% If you use BibTeX in apalike style, activate the following line:
\bibliographystyle{acm}
\title{CS-449 Project Milestone 2: Optimizing, Scaling, and Economics}
\author{
\textbf{Name}: xxx\\
\textbf{Sciper}: xxx\\
\textbf{Email:} xxx\\
\textbf{Name}: xxx\\
\textbf{Sciper}: xxx\\
\textbf{Email:} xxx\\
}
\begin{document}
\maketitle
\section{Optimizing with Breeze, a Linear Algebra Library}
\label{section:optimization}
\begin{itemize}
\item[\textbf{BR.1}] \BROne
\item [\textbf{BR.2}] \BRTwoOne
\BRTwoTwo
\BRTwoThree
\end{itemize}
\section{Parallel k-NN Computations with Replicated Ratings}
\begin{enumerate}
\item [\textbf{EK.1}] \EKOne
\item [\textbf{EK.2}] \EKTwo
\end{enumerate}
\section{Distributed Approximate k-NN}
\begin{enumerate}
\item [\textbf{AK.1}] \AKOne
\item [\textbf{AK.2}] \AKTwo
\item [\textbf{AK.3}] \AKThree
\end{enumerate}
\section{Economics}
\textit{Implement the computations for the different answers in the Economics.scala file. You don't need to provide unit tests for this question, nor written answers for these questions in your report.}
\end{document}
\ProvidesPackage{m2questions}[2022/03/11 v1.0]
% Breeze
\newcommand{\BROne}{
Reimplement the kNN predictor of Milestone 1 using the Breeze library and without using Spark. Using $k=10$ and \texttt{data/ml-100k/u2.base} for training, output the similarities between: (1) user $1$ and itself; (2) user $1$ and user $864$; (3) user $1$ and user $886$. Still using $k=10$, output the prediction for user 1 and item 1 ($p_{1,1}$), the prediction for user 327 and item 2 ($p_{327,2}$), and make sure that you obtain an MAE of $0.8287 \pm 0.0001$ on \texttt{data/ml-100k/u2.test}.
}
\newcommand{\BRTwoOne}{
Try making your implementation as fast as possible, both for computing all k-nearest neighbours and for computing the predictions and MAE on a test set. Your implementation should be based around \texttt{CSCMatrix}, but may involve conversions for individual operations. We will test your implementation on a secret test set. The teams with both a correct answer and the shortest time will receive more points.
}
\newcommand{\BRTwoTwo}{
Using $k=300$, compare the time for predicting all values and computing the MAE of \texttt{ml-100k/u2.test} to the one you obtained in Milestone 1. What is the speedup of your new implementation (as a ratio of $\frac{\textit{average time}_{old}}{\textit{average time}_{new}}$)? Use the same machine to measure the time for both versions and provide the answer in your report.
}
\newcommand{\BRTwoThree}{
Also ensure your implementation works with \texttt{data/ml-1m/rb.train} and \texttt{data/ml-1m/rb.test} since you will reuse it in the next questions.
}
% Parallel Exact Knn
\newcommand{\EKOne}{
Test your parallel implementation of k-NN for correctness with two workers. Using $k=10$ and \texttt{data/ml-100k/u2.base} for training, output the similarities between: (1) user $1$ and itself; (2) user $1$ and user $864$; (3) user $1$ and user $886$. Still using $k=10$, output the prediction for user 1 and item 1 ($p_{1,1}$), the prediction for user 327 and item 2 ($p_{327,2}$), and make sure that you obtain an MAE of $0.8287 \pm 0.0001$ on \texttt{data/ml-100k/u2.test}
}
\newcommand{\EKTwo}{
Measure and report the combined \textit{k-NN} and \textit{prediction} time when using 1, 2, 4 workers, $k=300$, and \texttt{ml-1m/rb.train} for training and \texttt{ml-1m/rb.test} for test, on the cluster (or a machine with at least 4 physical cores). Perform 3 measurements for each experiment and report the average and standard-deviation total time, including training, making predictions, and computing the MAE. Do you observe a speedup? Does this speedup grow linearly with the number of executors, i.e. is the running time $X$ times faster when using $X$ executors compared to using a single executor? Answer both questions in your report.
}
% Approximate Knn
\newcommand{\AKOne}{
Implement the approximate k-NN using your previous breeze implementation and Spark's RDDs. Using the partitioner of the template with 10 partitions and 2 replications, $k=10$, and \texttt{data/ml-100k/u2.base} for training, output the similarities of the approximate k-NN between user $1$ and the following users: $1,864,344,16,334,2$.
}
\newcommand{\AKTwo}{
Vary the number of partitions in which a given user appears. For the \texttt{data/ml-100k/u2.base} training set, partitioned equally between 10 workers, report the relationship between the level of replication (1,2,3,4,6,8) and the MAE you obtain on the \texttt{data/ml-100k/u2.test} test set. What is the minimum level of replication such that the MAE is still lower than the baseline predictor of Milestone 1 (MAE of 0.7604), when using $k=300$? Does this reduce the number of similarity computations compared to an exact k-NN? What is the ratio? Answer both questions in your report.
}
\newcommand{\AKThree}{
Measure and report the time required by your approximate \textit{k-NN} implementation, including both training on \texttt{data/ml-1m/rb.train} and computing the MAE on the test set \texttt{data/ml-1m/rb.test}, using $k=300$ on 8 partitions with a replication factor of 1 when using 1, 2, 4 workers. Perform each experiment 3 times and report the average and standard-deviation. Do you observe a speedup compared to the parallel (exact) k-NN with replicated ratings for the same number of workers?
}
% Economics
\newcommand{\EOne}{
What is the minimum number of days of renting to make buying the ICC.M7 less expensive, excluding any operating costs such as electricity and maintenance? Round up to the nearest integer.
}
\newcommand{\ETwoOne}{
After how many days of renting a container, is the cost higher than buying and running 4 Raspberry Pis? (1) Assuming optimistically no maintenance at minimum power usage for RPis, and (2) no maintenance at maximum power usage for RPis, to obtain a likely range. (Round up to the nearest integer in each case).
}
\newcommand{\ETwoTwo}{
Assume a single processor for the container and an equivalent amount of total RAM as the 4 Raspberry Pis. Also provide unrounded intermediary results for (1) Container Daily Cost, (2) 4 RPis (Idle) Daily Electricity Cost, (3) 4 RPis (Computing) Daily Electricity Cost.
}
\newcommand{\EThree}{
For the same buying price as an ICC.M7, how many Raspberry Pis can you get (floor the result to remove the decimal)? Assuming perfect scaling, would you obtain a larger overall throughput and RAM from these? If so, by how much? Compute the ratios using the previous floored number of RPis, but do not round the final results.
}
\ No newline at end of file
File added
# Milestone Description
[Milestone-2.pdf](./Milestone-2.pdf)
Note: Section 'Updates' lists the updates since the original release of the Milestone.
Mu has prepared a report template for your convenience here: [Report Template](./Milestone-2-QA-template.tex).
# Dependencies
````
sbt >= 1.4.7
openjdk@8
````
Should be available by default on ````iccluster028.iccluster.epfl.ch````. Otherwise, refer to each project installation instructions. Prefer working locally on your own machine, you will have less interference in your measurements from other students.
If you work on ````iccluster028.iccluster.epfl.ch````, you need to modify the PATH by default by adding the following line in ````~/.bashrc````:
````
export PATH=$PATH:/opt/sbt/sbt/bin
````
If you have multiple installations of openjdk, you need to specify the one to use as JAVA_HOME, e.g. on OSX with
openjdk@8 installed through Homebrew, you would do:
````
export JAVA_HOME="/usr/local/Cellar/openjdk@8/1.8.0+282";
````
# Dataset
Download [data-m2.zip](https://gitlab.epfl.ch/sacs/cs-449-sds-public/project/dataset/-/raw/main/data-m2.zip).
Unzip:
````
> unzip data-m2.zip
````
It should unzip into ````data/```` by default. If not, manually move ````ml-100k```` and ````ml-1m```` into ````data/````.
# Repository Structure
````src/main/scala/shared/predictions.scala````:
All the functionalities of your code for all questions should be defined there.
This code should then be used in the following applications and tests.
## Applications
1. ````src/main/scala/optimizing/Optimizing.scala````: Output answers to questions **BR.X**.
2. ````src/main/scala/distributed/Exact.scala````: Output answers to questions **EK.X**.
3. ````src/main/scala/distributed/Approximate.scala````: Output answers to questions **AK.X**.
4. ````src/main/scala/economics/Economics.scala````: Output answers to questions **E.X**
Applications are separate from tests to make it easier to test with different
inputs and permit outputting your answers and timings in JSON format for easier
grading.
## Unit Tests
Corresponding unit tests for each application (except Economics.scala):
````
src/test/scala/optimizing/OptimizingTests.scala
src/test/scala/distributed/ExactTests.scala
src/test/scala/distributed/ApproximateTests.scala
````
Your tests should demonstrate how to call your code to obtain the answers of
the applications, and should make exactly the same calls as for the
applications above. This structure intentionally encourages you to put as
little as possible functionality in the application. This also gives the TA a
clear and regular structure to check its correctness.
# Usage
## Execute unit tests
````
sbt "testOnly test.AllTests"
````
You should fill all tests and ensure they all succeed prior to submission.
## Run applications
### Optimizing
````
sbt "runMain scaling.Optimizing --train data/ml-100k/u2.base --test data/ml-100k/u2.test --json optimizing-100k.json --master local[1] --users 943 --movies 1682"
````
### Parallel Exact KNN
````
sbt "runMain distributed.Exact --train data/ml-100k/u2.base --test data/ml-100k/u2.test --json exact-100k-4.json --k 10 --master local[4] --users 943 --movies 1682"
````
### Approximate KNN
````
sbt "runMain distributed.Approximate --train data/ml-100k/u2.base --test data/ml-100k/u2.test --json approximate-100k-4-k10-r2.json --k 10 --master local[4] --users 943 --movies 1682 --partitions 10 --replication 2"
````
### Economics
````
sbt "runMain economics.Economics --json economics.json"
````
## Time applications
For all the previous applications, you can set the number of measurements for timings by adding the following option ````--num_measurements X```` where X is an integer. The default value is ````0````.
## IC Cluster
Test your application locally as much as possible and only test on the iccluster
once everything works, to keep the cluster and the driver node maximally available
for other students.
### Assemble Application for Spark Submit
````sbt clean````: clean up temporary files and previous assembly packages.
````sbt assembly````: create a new jar
````target/scala-2.11/m2_yourid-assembly-1.0.jar```` that can be used with
````spark-submit````.
Prefer packaging your application locally and upload the tar archive of your application
before running on cluster.
### Upload jar on Cluster
````
scp target/scala-2.11/m2_yourid-assembly-1.0.jar <username>@iccluster028.iccluster.epfl.ch:~
````
### Run on Cluster
See [config.sh](./config.sh) for HDFS paths to pre-uploaded train and test datasets to replace TRAIN and TEST, like in the example commands below:
#### When using ML-100k
````
spark-submit --class distributed.Exact --master yarn --conf "spark.dynamicAllocation.enabled=false" --num-executors 1 m2_yourid-assembly-1.0.jar --json exact-100k-1.json --train $ML100Ku2base --test $ML100Ku2test
````
#### When using ML-1m
````
spark-submit --class distributed.Exact --master yarn --conf "spark.dynamicAllocation.enabled=false" --num-executors 1 m2_yourid-assembly-1.0.jar --json exact-1m-1.json --train $ML1Mrbtrain --test $ML1Mrbtest --separator :: --k 300 --users 6040 --movies 3952
````
In order to keep results obtained with different parameters in different .json files, simply modify the corresponding parameter ("--json") passed and the values. For instance, with ```--num-executors 4``` : ```--json exact-1m-4.json```.
Note that when changing from ML-100k to ML-1M, the parameter ```--separator ::``` should be added, and the number of users and movies should be modified.
## Grading scripts
We will use the following scripts to grade your submission:
1. ````./test.sh````: Run all unit tests.
2. ````./run.sh````: Run all applications without timing measurements.
3. ````./time.sh````: Run all timing measurements.
All scripts will produce execution logs in the ````logs````
directory, including answers produced in the JSON format. Logs directories are
in the format ````logs/<scriptname>-<datetime>-<machine>/```` and include at
least an execution log ````log.txt```` as well as possible JSON outputs from
applications.
Ensure all scripts run correctly locally before submitting.
## Submission
Steps:
1. Create a zip archive with all your code within ````src/````, as well as your report: ````zip sciper1-sciper2.zip -r src/ report.pdf````
2. Submit ````sciper1-sciper2.zip```` the TA for grading on
https://cs449-submissions.epfl.ch:8083/m2 using the passcode you have previously received by email.
# References
Essential sbt: https://www.scalawilliam.com/essential-sbt/
Explore Spark Interactively (supports autocompletion with tabs!): https://spark.apache.org/docs/latest/quick-start.html
Scallop Argument Parsing: https://github.com/scallop/scallop/wiki
Spark Resilient Distributed Dataset (RDD): https://spark.apache.org/docs/3.0.1/api/scala/org/apache/spark/rdd/RDD.html
# Credits
Erick Lavoie (Design, Implementation, Tests)
Athanasios Xygkis (Requirements, Tests)
name := "m2_yourid"
version := "1.0"
libraryDependencies += "org.rogach" %% "scallop" % "4.0.2"
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.4.7"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.4.7"
libraryDependencies += "org.scalanlp" %% "breeze" % "0.13.2"
libraryDependencies += "org.scalanlp" %% "breeze-natives" % "0.13.2"
libraryDependencies += "org.scalatest" %% "scalatest" % "3.2.0" % Test
scalaVersion in ThisBuild := "2.13.3"
libraryDependencies += "com.lihaoyi" %% "ujson" % "1.5.0"
scalaVersion in ThisBuild := "2.11.12"
enablePlugins(JavaAppPackaging)
logBuffered in Test := false
test in assembly := {}
assemblyMergeStrategy in assembly := {
case PathList("META-INF", xs @ _*) => MergeStrategy.discard
case x => MergeStrategy.first
}
if [ $(hostname) == 'iccluster028' ];
then
ICCLUSTER=hdfs://iccluster028.iccluster.epfl.ch:8020
export ML100Ku2base=$ICCLUSTER/cs449/data/ml-100k/u2.base;
export ML100Ku2test=$ICCLUSTER/cs449/data/ml-100k/u2.test;
export ML100Kudata=$ICCLUSTER/cs449/data/ml-100k/u.data;
export ML1Mrbtrain=$ICCLUSTER/cs449/data/ml-1m/rb.train;
export ML1Mrbtest=$ICCLUSTER/cs449/data/ml-1m/rb.test;
export SPARKMASTER='yarn'
else
export ML100Ku2base=data/ml-100k/u2.base;
export ML100Ku2test=data/ml-100k/u2.test;
export ML100Kudata=data/ml-100k/u.data;
export ML1Mrbtrain=data/ml-1m/rb.train;
export ML1Mrbtest=data/ml-1m/rb.test;
export SPARKMASTER='local[4]'
fi;
id,title,rating
1,Toy Story (1995),5
2,GoldenEye (1995),3
3,Four Rooms (1995),
4,Get Shorty (1995),
5,Copycat (1995),
6,Shanghai Triad (Yao a yao yao dao waipo qiao) (1995),
7,Twelve Monkeys (1995),
8,Babe (1995),
9,Dead Man Walking (1995),
10,Richard III (1995),
11,Seven (Se7en) (1995),
12,Usual Suspects,
13,Mighty Aphrodite (1995),
14,Postino,
15,Mr. Holland's Opus (1995),
16,French Twist (Gazon maudit) (1995),
17,From Dusk Till Dawn (1996),
18,White Balloon,
19,Antonia's Line (1995),
20,Angels and Insects (1995),
21,Muppet Treasure Island (1996),
22,Braveheart (1995),3
23,Taxi Driver (1976),
24,Rumble in the Bronx (1995),
25,Birdcage,
26,Brothers McMullen,
27,Bad Boys (1995),
28,Apollo 13 (1995),3
29,Batman Forever (1995),
30,Belle de jour (1967),
31,Crimson Tide (1995),
32,Crumb (1994),
33,Desperado (1995),
34,Doom Generation,
35,Free Willy 2: The Adventure Home (1995),
36,Mad Love (1995),
37,Nadja (1994),
38,Net,
39,Strange Days (1995),
40,To Wong Foo,
41,Billy Madison (1995),
42,Clerks (1994),
43,Disclosure (1994),
44,Dolores Claiborne (1994),
45,Eat Drink Man Woman (1994),
46,Exotica (1994),
47,Ed Wood (1994),
48,Hoop Dreams (1994),
49,I.Q. (1994),
50,Star Wars (1977),4
51,Legends of the Fall (1994),
52,Madness of King George,
53,Natural Born Killers (1994),
54,Outbreak (1995),
55,Professional,
56,Pulp Fiction (1994),5
57,Priest (1994),
58,Quiz Show (1994),
59,Three Colors: Red (1994),
60,Three Colors: Blue (1993),
61,Three Colors: White (1994),
62,Stargate (1994),
63,Santa Clause,
64,Shawshank Redemption,5
65,What's Eating Gilbert Grape (1993),
66,While You Were Sleeping (1995),
67,Ace Ventura: Pet Detective (1994),
68,Crow,
69,Forrest Gump (1994),5
70,Four Weddings and a Funeral (1994),
71,Lion King,5
72,Mask,
73,Maverick (1994),
74,Faster Pussycat! Kill! Kill! (1965),
75,Brother Minister: The Assassination of Malcolm X (1994),
76,Carlito's Way (1993),
77,Firm,
78,Free Willy (1993),
79,Fugitive,
80,Hot Shots! Part Deux (1993),
81,Hudsucker Proxy,
82,Jurassic Park (1993),3
83,Much Ado About Nothing (1993),
84,Robert A. Heinlein's The Puppet Masters (1994),
85,Ref,
86,Remains of the Day,
87,Searching for Bobby Fischer (1993),
88,Sleepless in Seattle (1993),
89,Blade Runner (1982),3
90,So I Married an Axe Murderer (1993),
91,Nightmare Before Christmas,
92,True Romance (1993),
93,Welcome to the Dollhouse (1995),
94,Home Alone (1990),
95,Aladdin (1992),4
96,Terminator 2: Judgment Day (1991),5
97,Dances with Wolves (1990),
98,Silence of the Lambs,
99,Snow White and the Seven Dwarfs (1937),1
100,Fargo (1996),
101,Heavy Metal (1981),
102,Aristocats,
103,All Dogs Go to Heaven 2 (1996),
104,Theodore Rex (1995),
105,Sgt. Bilko (1996),
106,Diabolique (1996),
107,Moll Flanders (1996),
108,Kids in the Hall: Brain Candy (1996),
109,Mystery Science Theater 3000: The Movie (1996),
110,Operation Dumbo Drop (1995),
111,Truth About Cats & Dogs,
112,Flipper (1996),
113,Horseman on the Roof,
114,Wallace & Gromit: The Best of Aardman Animation (1996),
115,Haunted World of Edward D. Wood Jr.,
116,Cold Comfort Farm (1995),
117,Rock,
118,Twister (1996),
119,Maya Lin: A Strong Clear Vision (1994),
120,Striptease (1996),
121,Independence Day (ID4) (1996),1
122,Cable Guy,1
123,Frighteners,
124,Lone Star (1996),
125,Phenomenon (1996),
126,Spitfire Grill,
127,Godfather,5
128,Supercop (1992),
129,Bound (1996),
130,Kansas City (1996),
131,Breakfast at Tiffany's (1961),
132,Wizard of Oz,
133,Gone with the Wind (1939),
134,Citizen Kane (1941),
135,2001: A Space Odyssey (1968),
136,Mr. Smith Goes to Washington (1939),
137,Big Night (1996),
138,D3: The Mighty Ducks (1996),2
139,Love Bug,
140,Homeward Bound: The Incredible Journey (1993),
141,20,
142,Bedknobs and Broomsticks (1971),
143,Sound of Music,4
144,Die Hard (1988),3
145,Lawnmower Man,
146,Unhook the Stars (1996),
147,Long Kiss Goodnight,
148,Ghost and the Darkness,
149,Jude (1996),
150,Swingers (1996),
151,Willy Wonka and the Chocolate Factory (1971),4
152,Sleeper (1973),
153,Fish Called Wanda,
154,Monty Python's Life of Brian (1979),5
155,Dirty Dancing (1987),3
156,Reservoir Dogs (1992),5
157,Platoon (1986),
158,Weekend at Bernie's (1989),
159,Basic Instinct (1992),
160,Glengarry Glen Ross (1992),
161,Top Gun (1986),3
162,On Golden Pond (1981),
163,Return of the Pink Panther,4
164,Abyss,2
165,Jean de Florette (1986),
166,Manon of the Spring (Manon des sources) (1986),
167,Private Benjamin (1980),
168,Monty Python and the Holy Grail (1974),5
169,Wrong Trousers,
170,Cinema Paradiso (1988),
171,Delicatessen (1991),
172,Empire Strikes Back,2
173,Princess Bride,
174,Raiders of the Lost Ark (1981),
175,Brazil (1985),
176,Aliens (1986),
177,"The Good the Bad and the Ugly",5
178,12 Angry Men (1957),
179,Clockwork Orange,4
180,Apocalypse Now (1979),3
181,Return of the Jedi (1983),3
182,GoodFellas (1990),
183,Alien (1979),5
184,Army of Darkness (1993),2
185,Psycho (1960),4
186,Blues Brothers,
187,Godfather: Part II,4
188,Full Metal Jacket (1987),5
189,Grand Day Out,
190,Henry V (1989),
191,Amadeus (1984),3
192,Raging Bull (1980),
193,Right Stuff,
194,Sting,
195,Terminator,
196,Dead Poets Society (1989),5
197,Graduate,
198,Nikita (La Femme Nikita) (1990),
199,Bridge on the River Kwai,
200,Shining,
201,Evil Dead II (1987),
202,Groundhog Day (1993),5
203,Unforgiven (1992),
204,Back to the Future (1985),2
205,Patton (1970),
206,Akira (1988),
207,Cyrano de Bergerac (1990),
208,Young Frankenstein (1974),
209,This Is Spinal Tap (1984),
210,Indiana Jones and the Last Crusade (1989),
211,M*A*S*H (1970),
212,Unbearable Lightness of Being,
213,Room with a View,
214,Pink Floyd - The Wall (1982),4
215,Field of Dreams (1989),
216,When Harry Met Sally... (1989),
217,Bram Stoker's Dracula (1992),
218,Cape Fear (1991),
219,Nightmare on Elm Street,
220,Mirror Has Two Faces,
221,Breaking the Waves (1996),
222,Star Trek: First Contact (1996),
223,Sling Blade (1996),
224,Ridicule (1996),
225,101 Dalmatians (1996),
226,Die Hard 2 (1990),
227,Star Trek VI: The Undiscovered Country (1991),
228,Star Trek: The Wrath of Khan (1982),
229,Star Trek III: The Search for Spock (1984),
230,Star Trek IV: The Voyage Home (1986),
231,Batman Returns (1992),
232,Young Guns (1988),
233,Under Siege (1992),
234,Jaws (1975),
235,Mars Attacks! (1996),2
236,Citizen Ruth (1996),
237,Jerry Maguire (1996),3
238,Raising Arizona (1987),
239,Sneakers (1992),
240,Beavis and Butt-head Do America (1996),
241,Last of the Mohicans,
242,Kolya (1996),
243,Jungle2Jungle (1997),
244,Smilla's Sense of Snow (1997),
245,Devil's Own,
246,Chasing Amy (1997),
247,Turbo: A Power Rangers Movie (1997),
248,Grosse Pointe Blank (1997),
249,Austin Powers: International Man of Mystery (1997),
250,Fifth Element,2
251,Shall We Dance? (1996),
252,Lost World: Jurassic Park,
253,Pillow Book,
254,Batman & Robin (1997),
255,My Best Friend's Wedding (1997),
256,When the Cats Away (Chacun cherche son chat) (1996),
257,Men in Black (1997),2
258,Contact (1997),3
259,George of the Jungle (1997),3
260,Event Horizon (1997),
261,Air Bud (1997),
262,In the Company of Men (1997),
263,Steel (1997),
264,Mimic (1997),
265,Hunt for Red October,
266,Kull the Conqueror (1997),
267,unknown,
268,Chasing Amy (1997),
269,Full Monty,
270,Gattaca (1997),3
271,Starship Troopers (1997),
272,Good Will Hunting (1997),5
273,Heat (1995),
274,Sabrina (1995),
275,Sense and Sensibility (1995),
276,Leaving Las Vegas (1995),
277,Restoration (1995),
278,Bed of Roses (1996),
279,Once Upon a Time... When We Were Colored (1995),
280,Up Close and Personal (1996),
281,River Wild,
282,Time to Kill,
283,Emma (1996),
284,Tin Cup (1996),
285,Secrets & Lies (1996),
286,English Patient,
287,Marvin's Room (1996),
288,Scream (1996),
289,Evita (1996),
290,Fierce Creatures (1997),
291,Absolute Power (1997),
292,Rosewood (1997),
293,Donnie Brasco (1997),
294,Liar Liar (1997),2
295,Breakdown (1997),
296,Promesse,
297,Ulee's Gold (1997),
298,Face/Off (1997),
299,Hoodlum (1997),
300,Air Force One (1997),
301,In & Out (1997),
302,L.A. Confidential (1997),
303,Ulee's Gold (1997),
304,Fly Away Home (1996),
305,Ice Storm,
306,Mrs. Brown (Her Majesty,
307,Devil's Advocate,
308,FairyTale: A True Story (1997),
309,Deceiver (1997),
310,Rainmaker,
311,Wings of the Dove,
312,Midnight in the Garden of Good and Evil (1997),
313,Titanic (1997),3
314,3 Ninjas: High Noon At Mega Mountain (1998),
315,Apt Pupil (1998),
316,As Good As It Gets (1997),
317,In the Name of the Father (1993),
318,Schindler's List (1993),
319,Everyone Says I Love You (1996),
320,Paradise Lost: The Child Murders at Robin Hood Hills (1996),
321,Mother (1996),
322,Murder at 1600 (1997),
323,Dante's Peak (1997),
324,Lost Highway (1997),
325,Crash (1996),
326,G.I. Jane (1997),
327,Cop Land (1997),
328,Conspiracy Theory (1997),
329,Desperate Measures (1998),
330,187 (1997),
331,Edge,
332,Kiss the Girls (1997),
333,Game,
334,U Turn (1997),
335,How to Be a Player (1997),
336,Playing God (1997),
337,House of Yes,
338,Bean (1997),
339,Mad City (1997),
340,Boogie Nights (1997),
341,Critical Care (1997),
342,Man Who Knew Too Little,
343,Alien: Resurrection (1997),
344,Apostle,
345,Deconstructing Harry (1997),
346,Jackie Brown (1997),
347,Wag the Dog (1997),
348,Desperate Measures (1998),
349,Hard Rain (1998),
350,Fallen (1998),
351,Prophecy II,
352,Spice World (1997),
353,Deep Rising (1998),
354,Wedding Singer,
355,Sphere (1998),
356,Client,
357,One Flew Over the Cuckoo's Nest (1975),
358,Spawn (1997),
359,Assignment,
360,Wonderland (1997),
361,Incognito (1997),
362,Blues Brothers 2000 (1998),1
363,Sudden Death (1995),
364,Ace Ventura: When Nature Calls (1995),1
365,Powder (1995),
366,Dangerous Minds (1995),
367,Clueless (1995),
368,Bio-Dome (1996),
369,Black Sheep (1996),
370,Mary Reilly (1996),
371,Bridges of Madison County,
372,Jeffrey (1995),
373,Judge Dredd (1995),
374,Mighty Morphin Power Rangers: The Movie (1995),1
375,Showgirls (1995),
376,Houseguest (1994),
377,Heavyweights (1994),
378,Miracle on 34th Street (1994),
379,Tales From the Crypt Presents: Demon Knight (1995),
380,Star Trek: Generations (1994),
381,Muriel's Wedding (1994),
382,Adventures of Priscilla,
383,Flintstones,
384,Naked Gun 33 1/3: The Final Insult (1994),3
385,True Lies (1994),2
386,Addams Family Values (1993),
387,Age of Innocence,
388,Beverly Hills Cop III (1994),
389,Black Beauty (1994),
390,Fear of a Black Hat (1993),
391,Last Action Hero (1993),
392,Man Without a Face,
393,Mrs. Doubtfire (1993),
394,Radioland Murders (1994),
395,Robin Hood: Men in Tights (1993),
396,Serial Mom (1994),
397,Striking Distance (1993),
398,Super Mario Bros. (1993),1
399,Three Musketeers,
400,Little Rascals,
401,Brady Bunch Movie,
402,Ghost (1990),
403,Batman (1989),
404,Pinocchio (1940),
405,Mission: Impossible (1996),3
406,Thinner (1996),
407,Spy Hard (1996),
408,Close Shave,
409,Jack (1996),
410,Kingpin (1996),
411,Nutty Professor,
412,Very Brady Sequel,
413,Tales from the Crypt Presents: Bordello of Blood (1996),
414,My Favorite Year (1982),
415,Apple Dumpling Gang,
416,Old Yeller (1957),
417,Parent Trap,
418,Cinderella (1950),
419,Mary Poppins (1964),
420,Alice in Wonderland (1951),
421,William Shakespeare's Romeo and Juliet (1996),
422,Aladdin and the King of Thieves (1996),
423,E.T. the Extra-Terrestrial (1982),
424,Children of the Corn: The Gathering (1996),
425,Bob Roberts (1992),
426,Transformers: The Movie,
427,To Kill a Mockingbird (1962),
428,Harold and Maude (1971),
429,Day the Earth Stood Still,
430,Duck Soup (1933),
431,Highlander (1986),2
432,Fantasia (1940),
433,Heathers (1989),
434,Forbidden Planet (1956),
435,Butch Cassidy and the Sundance Kid (1969),
436,American Werewolf in London,
437,Amityville 1992: It's About Time (1992),
438,Amityville 3-D (1983),
439,Amityville: A New Generation (1993),
440,Amityville II: The Possession (1982),
441,Amityville Horror,
442,Amityville Curse,
443,Birds,
444,Blob,
445,Body Snatcher,
446,Burnt Offerings (1976),
447,Carrie (1976),
448,Omen,
449,Star Trek: The Motion Picture (1979),
450,Star Trek V: The Final Frontier (1989),
451,Grease (1978),
452,Jaws 2 (1978),
453,Jaws 3-D (1983),
454,Bastard Out of Carolina (1996),
455,Jackie Chan's First Strike (1996),
456,Beverly Hills Ninja (1997),
457,Free Willy 3: The Rescue (1997),
458,Nixon (1995),
459,Cry,
460,Crossing Guard,
461,Smoke (1995),
462,Like Water For Chocolate (Como agua para chocolate) (1992),
463,Secret of Roan Inish,
464,Vanya on 42nd Street (1994),
465,Jungle Book,
466,Red Rock West (1992),
467,Bronx Tale,
468,Rudy (1993),
469,Short Cuts (1993),
470,Tombstone (1993),
471,Courage Under Fire (1996),
472,Dragonheart (1996),
473,James and the Giant Peach (1996),
474,Dr. Strangelove or: How I Learned to Stop Worrying and Love the Bomb (1963),
475,Trainspotting (1996),
476,First Wives Club,
477,Matilda (1996),
478,Philadelphia Story,
479,Vertigo (1958),
480,North by Northwest (1959),
481,Apartment,
482,Some Like It Hot (1959),
483,Casablanca (1942),
484,Maltese Falcon,
485,My Fair Lady (1964),
486,Sabrina (1954),
487,Roman Holiday (1953),
488,Sunset Blvd. (1950),
489,Notorious (1946),
490,To Catch a Thief (1955),
491,Adventures of Robin Hood,
492,East of Eden (1955),
493,Thin Man,
494,His Girl Friday (1940),
495,Around the World in 80 Days (1956),
496,It's a Wonderful Life (1946),
497,Bringing Up Baby (1938),
498,African Queen,
499,Cat on a Hot Tin Roof (1958),
500,Fly Away Home (1996),
501,Dumbo (1941),3
502,Bananas (1971),
503,Candidate,
504,Bonnie and Clyde (1967),
505,Dial M for Murder (1954),
506,Rebel Without a Cause (1955),
507,Streetcar Named Desire,
508,People vs. Larry Flynt,
509,My Left Foot (1989),
510,Magnificent Seven,
511,Lawrence of Arabia (1962),
512,Wings of Desire (1987),
513,Third Man,
514,Annie Hall (1977),
515,Boot,
516,Local Hero (1983),
517,Manhattan (1979),
518,Miller's Crossing (1990),
519,Treasure of the Sierra Madre,
520,Great Escape,
521,Deer Hunter,
522,Down by Law (1986),
523,Cool Hand Luke (1967),
524,Great Dictator,
525,Big Sleep,
526,Ben-Hur (1959),
527,Gandhi (1982),
528,Killing Fields,
529,My Life as a Dog (Mitt liv som hund) (1985),
530,Man Who Would Be King,
531,Shine (1996),
532,Kama Sutra: A Tale of Love (1996),
533,Daytrippers,
534,Traveller (1997),
535,Addicted to Love (1997),
536,Ponette (1996),
537,My Own Private Idaho (1991),
538,Anastasia (1997),
539,Mouse Hunt (1997),
540,Money Train (1995),
541,Mortal Kombat (1995),1
542,Pocahontas (1995),
543,Misérables,
544,Things to Do in Denver when You're Dead (1995),
545,Vampire in Brooklyn (1995),
546,Broken Arrow (1996),
547,Young Poisoner's Handbook,
548,NeverEnding Story III,
549,Rob Roy (1995),
550,Die Hard: With a Vengeance (1995),
551,Lord of Illusions (1995),
552,Species (1995),
553,Walk in the Clouds,
554,Waterworld (1995),
555,White Man's Burden (1995),
556,Wild Bill (1995),
557,Farinelli: il castrato (1994),
558,Heavenly Creatures (1994),
559,Interview with the Vampire (1994),
560,Kid in King Arthur's Court,
561,Mary Shelley's Frankenstein (1994),
562,Quick and the Dead,
563,Stephen King's The Langoliers (1995),
564,Tales from the Hood (1995),
565,Village of the Damned (1995),
566,Clear and Present Danger (1994),
567,Wes Craven's New Nightmare (1994),
568,Speed (1994),
569,Wolf (1994),
570,Wyatt Earp (1994),
571,Another Stakeout (1993),
572,Blown Away (1994),
573,Body Snatchers (1993),
574,Boxing Helena (1993),
575,City Slickers II: The Legend of Curly's Gold (1994),
576,Cliffhanger (1993),
577,Coneheads (1993),
578,Demolition Man (1993),
579,Fatal Instinct (1993),
580,Englishman Who Went Up a Hill,
581,Kalifornia (1993),
582,Piano,
583,Romeo Is Bleeding (1993),
584,Secret Garden,
585,Son in Law (1993),
586,Terminal Velocity (1994),
587,Hour of the Pig,
588,Beauty and the Beast (1991),
589,Wild Bunch,
590,Hellraiser: Bloodline (1996),
591,Primal Fear (1996),
592,True Crime (1995),
593,Stalingrad (1993),
594,Heavy (1995),
595,Fan,
596,Hunchback of Notre Dame,
597,Eraser (1996),
598,Big Squeeze,
599,Police Story 4: Project S (Chao ji ji hua) (1993),
600,Daniel Defoe's Robinson Crusoe (1996),
601,For Whom the Bell Tolls (1943),
602,American in Paris,
603,Rear Window (1954),
604,It Happened One Night (1934),
605,Meet Me in St. Louis (1944),
606,All About Eve (1950),
607,Rebecca (1940),
608,Spellbound (1945),
609,Father of the Bride (1950),
610,Gigi (1958),
611,Laura (1944),
612,Lost Horizon (1937),
613,My Man Godfrey (1936),
614,Giant (1956),
615,39 Steps,
616,Night of the Living Dead (1968),
617,Blue Angel,
618,Picnic (1955),
619,Extreme Measures (1996),
620,Chamber,
621,Davy Crockett,
622,Swiss Family Robinson (1960),
623,Angels in the Outfield (1994),
624,Three Caballeros,
625,Sword in the Stone,
626,So Dear to My Heart (1949),
627,Robin Hood: Prince of Thieves (1991),
628,Sleepers (1996),
629,Victor/Victoria (1982),
630,Great Race,
631,Crying Game,
632,Sophie's Choice (1982),
633,Christmas Carol,
634,Microcosmos: Le peuple de l'herbe (1996),
635,Fog,
636,Escape from New York (1981),
637,Howling,
638,Return of Martin Guerre,
639,Tin Drum,
640,Cook the Thief His Wife & Her Lover,
641,Paths of Glory (1957),
642,Grifters,
643,The Innocent (1994),
644,Thin Blue Line,
645,Paris Is Burning (1990),
646,Once Upon a Time in the West (1969),
647,Ran (1985),
648,Quiet Man,
649,Once Upon a Time in America (1984),
650,Seventh Seal,
651,Glory (1989),
652,Rosencrantz and Guildenstern Are Dead (1990),
653,Touch of Evil (1958),
654,Chinatown (1974),
655,Stand by Me (1986),
656,M (1931),
657,Manchurian Candidate,
658,Pump Up the Volume (1990),
659,Arsenic and Old Lace (1944),
660,Fried Green Tomatoes (1991),
661,High Noon (1952),
662,Somewhere in Time (1980),
663,Being There (1979),
664,Paris,
665,Alien 3 (1992),
666,Blood For Dracula (Andy Warhol's Dracula) (1974),
667,Audrey Rose (1977),
668,Blood Beach (1981),
669,Body Parts (1991),
670,Body Snatchers (1993),
671,Bride of Frankenstein (1935),
672,Candyman (1992),1
673,Cape Fear (1962),
674,Cat People (1982),
675,Nosferatu (Nosferatu,
676,Crucible,
677,Fire on the Mountain (1996),
678,Volcano (1997),
679,Conan the Barbarian (1981),5
680,Kull the Conqueror (1997),
681,Wishmaster (1997),
682,I Know What You Did Last Summer (1997),
683,Rocket Man (1997),
684,In the Line of Fire (1993),
685,Executive Decision (1996),
686,Perfect World,
687,McHale's Navy (1997),
688,Leave It to Beaver (1997),
689,Jackal,
690,Seven Years in Tibet (1997),
691,Dark City (1998),3
692,American President,
693,Casino (1995),
694,Persuasion (1995),
695,Kicking and Screaming (1995),
696,City Hall (1996),
697,Basketball Diaries,
698,Browning Version,
699,Little Women (1994),
700,Miami Rhapsody (1995),
701,Wonderful,
702,Barcelona (1994),
703,Widows' Peak (1994),
704,House of the Spirits,
705,Singin' in the Rain (1952),
706,Bad Moon (1996),
707,Enchanted April (1991),
708,Sex,
709,Strictly Ballroom (1992),
710,Better Off Dead... (1985),
711,Substance of Fire,
712,Tin Men (1987),
713,Othello (1995),
714,Carrington (1995),
715,To Die For (1995),
716,Home for the Holidays (1995),
717,Juror,
718,In the Bleak Midwinter (1995),
719,Canadian Bacon (1994),
720,First Knight (1995),
721,Mallrats (1995),
722,Nine Months (1995),
723,Boys on the Side (1995),
724,Circle of Friends (1995),
725,Exit to Eden (1994),
726,Fluke (1995),
727,Immortal Beloved (1994),
728,Junior (1994),
729,Nell (1994),
730,Queen Margot (Reine Margot,
731,Corrina,
732,Dave (1993),
733,Go Fish (1994),
734,Made in America (1993),
735,Philadelphia (1993),
736,Shadowlands (1993),
737,Sirens (1994),
738,Threesome (1994),1
739,Pretty Woman (1990),
740,Jane Eyre (1996),
741,Last Supper,
742,Ransom (1996),
743,Crow: City of Angels,
744,Michael Collins (1996),
745,Ruling Class,
746,Real Genius (1985),
747,Benny & Joon (1993),
748,Saint,
749,MatchMaker,
750,Amistad (1997),4
751,Tomorrow Never Dies (1997),
752,Replacement Killers,
753,Burnt By the Sun (1994),
754,Red Corner (1997),
755,Jumanji (1995),
756,Father of the Bride Part II (1995),
757,Across the Sea of Time (1995),
758,Lawnmower Man 2: Beyond Cyberspace (1996),
759,Fair Game (1995),
760,Screamers (1995),
761,Nick of Time (1995),
762,Beautiful Girls (1996),
763,Happy Gilmore (1996),
764,If Lucy Fell (1996),
765,Boomerang (1992),
766,Man of the Year (1995),
767,Addiction,
768,Casper (1995),1
769,Congo (1995),
770,Devil in a Blue Dress (1995),
771,Johnny Mnemonic (1995),2
772,Kids (1995),
773,Mute Witness (1994),
774,Prophecy,
775,Something to Talk About (1995),
776,Three Wishes (1995),
777,Castle Freak (1995),
778,Don Juan DeMarco (1995),
779,Drop Zone (1994),
780,Dumb & Dumber (1994),
781,French Kiss (1995),
782,Little Odessa (1994),
783,Milk Money (1994),
784,Beyond Bedlam (1993),
785,Only You (1994),
786,Perez Family,
787,Roommates (1995),
788,Relative Fear (1994),
789,Swimming with Sharks (1995),
790,Tommy Boy (1995),
791,Baby-Sitters Club,
792,Bullets Over Broadway (1994),
793,Crooklyn (1994),
794,It Could Happen to You (1994),
795,Richie Rich (1994),
796,Speechless (1994),
797,Timecop (1994),
798,Bad Company (1995),
799,Boys Life (1995),
800,In the Mouth of Madness (1995),
801,Air Up There,
802,Hard Target (1993),
803,Heaven & Earth (1993),
804,Jimmy Hollywood (1994),
805,Manhattan Murder Mystery (1993),
806,Menace II Society (1993),
807,Poetic Justice (1993),
808,Program,
809,Rising Sun (1993),
810,Shadow,
811,Thirty-Two Short Films About Glenn Gould (1993),
812,Andre (1994),
813,Celluloid Closet,
814,Great Day in Harlem,
815,One Fine Day (1996),
816,Candyman: Farewell to the Flesh (1995),
817,Frisk (1995),
818,Girl 6 (1996),
819,Eddie (1996),
820,Space Jam (1996),
821,Mrs. Winterbourne (1996),
822,Faces (1968),
823,Mulholland Falls (1996),
824,Great White Hype,
825,Arrival,
826,Phantom,
827,Daylight (1996),
828,Alaska (1996),
829,Fled (1996),
830,Power 98 (1995),
831,Escape from L.A. (1996),
832,Bogus (1996),
833,Bulletproof (1996),
834,Halloween: The Curse of Michael Myers (1995),
835,Gay Divorcee,
836,Ninotchka (1939),
837,Meet John Doe (1941),
838,In the Line of Duty 2 (1987),
839,Loch Ness (1995),
840,Last Man Standing (1996),
841,Glimmer Man,
842,Pollyanna (1960),
843,Shaggy Dog,
844,Freeway (1996),
845,That Thing You Do! (1996),
846,To Gillian on Her 37th Birthday (1996),
847,Looking for Richard (1996),
848,Murder,
849,Days of Thunder (1990),
850,Perfect Candidate,
851,Two or Three Things I Know About Her (1966),
852,Bloody Child,
853,Braindead (1992),
854,Bad Taste (1987),
855,Diva (1981),
856,Night on Earth (1991),
857,Paris Was a Woman (1995),
858,Amityville: Dollhouse (1996),
859,April Fool's Day (1986),
860,Believers,
861,Nosferatu a Venezia (1986),
862,Jingle All the Way (1996),
863,Garden of Finzi-Contini,
864,My Fellow Americans (1996),
865,Ice Storm,
866,Michael (1996),
867,Whole Wide World,
868,Hearts and Minds (1996),
869,Fools Rush In (1997),
870,Touch (1997),
871,Vegas Vacation (1997),
872,Love Jones (1997),
873,Picture Perfect (1997),
874,Career Girls (1997),
875,She's So Lovely (1997),
876,Money Talks (1997),
877,Excess Baggage (1997),
878,That Darn Cat! (1997),
879,Peacemaker,
880,Soul Food (1997),
881,Money Talks (1997),
882,Washington Square (1997),
883,Telling Lies in America (1997),
884,Year of the Horse (1997),
885,Phantoms (1998),
886,Life Less Ordinary,
887,Eve's Bayou (1997),
888,One Night Stand (1997),
889,Tango Lesson,
890,Mortal Kombat: Annihilation (1997),
891,Bent (1997),
892,Flubber (1997),
893,For Richer or Poorer (1997),
894,Home Alone 3 (1997),
895,Scream 2 (1997),
896,Sweet Hereafter,
897,Time Tracers (1995),
898,Postman,
899,Winter Guest,
900,Kundun (1997),
901,Mr. Magoo (1997),
902,Big Lebowski,3
903,Afterglow (1997),
904,Ma vie en rose (My Life in Pink) (1997),
905,Great Expectations (1998),
906,Oscar & Lucinda (1997),
907,Vermin (1998),
908,Half Baked (1998),
909,Dangerous Beauty (1998),
910,Nil By Mouth (1997),
911,Twilight (1998),
912,U.S. Marshalls (1998),
913,Love and Death on Long Island (1997),
914,Wild Things (1998),
915,Primary Colors (1998),
916,Lost in Space (1998),
917,Mercury Rising (1998),
918,City of Angels (1998),
919,City of Lost Children,
920,Two Bits (1995),
921,Farewell My Concubine (1993),
922,Dead Man (1995),
923,Raise the Red Lantern (1991),
924,White Squall (1996),
925,Unforgettable (1996),
926,Down Periscope (1996),
927,Flower of My Secret,
928,Craft,
929,Harriet the Spy (1996),
930,Chain Reaction (1996),
931,Island of Dr. Moreau,
932,First Kid (1996),
933,Funeral,
934,Preacher's Wife,
935,Paradise Road (1997),
936,Brassed Off (1996),
937,Thousand Acres,
938,Smile Like Yours,
939,Murder in the First (1995),
940,Airheads (1994),
941,With Honors (1994),
942,What's Love Got to Do with It (1993),
943,Killing Zoe (1994),
944,Renaissance Man (1994),
945,Charade (1963),
946,Fox and the Hound,
947,Big Blue,
948,Booty Call (1997),
949,How to Make an American Quilt (1995),
950,Georgia (1995),
951,Indian in the Cupboard,
952,Blue in the Face (1995),
953,Unstrung Heroes (1995),
954,Unzipped (1995),
955,Before Sunrise (1995),
956,Nobody's Fool (1994),
957,Pushing Hands (1992),
958,To Live (Huozhe) (1994),
959,Dazed and Confused (1993),
960,Naked (1993),
961,Orlando (1993),
962,Ruby in Paradise (1993),
963,Some Folks Call It a Sling Blade (1993),
964,Month by the Lake,
965,Funny Face (1957),
966,Affair to Remember,
967,Little Lord Fauntleroy (1936),
968,Inspector General,
969,Winnie the Pooh and the Blustery Day (1968),
970,Hear My Song (1991),
971,Mediterraneo (1991),
972,Passion Fish (1992),
973,Grateful Dead (1995),
974,Eye for an Eye (1996),
975,Fear (1996),
976,Solo (1996),
977,Substitute,
978,Heaven's Prisoners (1996),
979,Trigger Effect,
980,Mother Night (1996),
981,Dangerous Ground (1997),
982,Maximum Risk (1996),
983,Rich Man's Wife,
984,Shadow Conspiracy (1997),
985,Blood & Wine (1997),
986,Turbulence (1997),
987,Underworld (1997),
988,Beautician and the Beast,
989,Cats Don't Dance (1997),
990,Anna Karenina (1997),
991,Keys to Tulsa (1997),
992,Head Above Water (1996),
993,Hercules (1997),
994,Last Time I Committed Suicide,
995,Kiss Me,
996,Big Green,
997,Stuart Saves His Family (1995),
998,Cabin Boy (1994),
999,Clean Slate (1994),
1000,Lightning Jack (1994),
1001,Stupids,
1002,Pest,
1003,That Darn Cat! (1997),
1004,Geronimo: An American Legend (1993),
1005,Double vie de Véronique,
1006,Until the End of the World (Bis ans Ende der Welt) (1991),
1007,Waiting for Guffman (1996),
1008,I Shot Andy Warhol (1996),
1009,Stealing Beauty (1996),
1010,Basquiat (1996),
1011,2 Days in the Valley (1996),
1012,Private Parts (1997),
1013,Anaconda (1997),
1014,Romy and Michele's High School Reunion (1997),
1015,Shiloh (1997),
1016,Con Air (1997),
1017,Trees Lounge (1996),
1018,Tie Me Up! Tie Me Down! (1990),
1019,Die xue shuang xiong (Killer,
1020,Gaslight (1944),
1021,8 1/2 (1963),
1022,Fast,
1023,Fathers' Day (1997),
1024,Mrs. Dalloway (1997),
1025,Fire Down Below (1997),
1026,Lay of the Land,
1027,Shooter,
1028,Grumpier Old Men (1995),
1029,Jury Duty (1995),
1030,Beverly Hillbillies,
1031,Lassie (1994),
1032,Little Big League (1994),
1033,Homeward Bound II: Lost in San Francisco (1996),
1034,Quest,
1035,Cool Runnings (1993),
1036,Drop Dead Fred (1991),
1037,Grease 2 (1982),
1038,Switchback (1997),
1039,Hamlet (1996),
1040,Two if by Sea (1996),
1041,Forget Paris (1995),
1042,Just Cause (1995),
1043,Rent-a-Kid (1995),
1044,Paper,
1045,Fearless (1993),
1046,Malice (1993),
1047,Multiplicity (1996),
1048,She's the One (1996),
1049,House Arrest (1996),
1050,Ghost and Mrs. Muir,
1051,Associate,
1052,Dracula: Dead and Loving It (1995),
1053,Now and Then (1995),
1054,Mr. Wrong (1996),
1055,Simple Twist of Fate,
1056,Cronos (1992),
1057,Pallbearer,
1058,War,
1059,Don't Be a Menace to South Central While Drinking Your Juice in the Hood (1996),
1060,Adventures of Pinocchio,
1061,Evening Star,
1062,Four Days in September (1997),
1063,Little Princess,
1064,Crossfire (1947),
1065,Koyaanisqatsi (1983),4
1066,Balto (1995),
1067,Bottle Rocket (1996),
1068,Star Maker,
1069,Amateur (1994),
1070,Living in Oblivion (1995),
1071,Party Girl (1995),
1072,Pyromaniac's Love Story,
1073,Shallow Grave (1994),
1074,Reality Bites (1994),
1075,Man of No Importance,
1076,Pagemaster,
1077,Love and a .45 (1994),
1078,Oliver & Company (1988),
1079,Joe's Apartment (1996),
1080,Celestial Clockwork (1994),
1081,Curdled (1996),
1082,Female Perversions (1996),
1083,Albino Alligator (1996),
1084,Anne Frank Remembered (1995),
1085,Carried Away (1996),
1086,It's My Party (1995),
1087,Bloodsport 2 (1995),
1088,Double Team (1997),
1089,Speed 2: Cruise Control (1997),
1090,Sliver (1993),
1091,Pete's Dragon (1977),
1092,Dear God (1996),
1093,Live Nude Girls (1995),
1094,Thin Line Between Love and Hate,
1095,High School High (1996),
1096,Commandments (1997),
1097,Hate (Haine,
1098,Flirting With Disaster (1996),
1099,Red Firecracker,
1100,What Happened Was... (1994),
1101,Six Degrees of Separation (1993),
1102,Two Much (1996),
1103,Trust (1990),
1104,C'est arrivé près de chez vous (1992),
1105,Firestorm (1998),
1106,Newton Boys,
1107,Beyond Rangoon (1995),
1108,Feast of July (1995),
1109,Death and the Maiden (1994),
1110,Tank Girl (1995),
1111,Double Happiness (1994),
1112,Cobb (1994),
1113,Mrs. Parker and the Vicious Circle (1994),
1114,Faithful (1996),
1115,Twelfth Night (1996),
1116,Mark of Zorro,
1117,Surviving Picasso (1996),
1118,Up in Smoke (1978),
1119,Some Kind of Wonderful (1987),
1120,I'm Not Rappaport (1996),
1121,Umbrellas of Cherbourg,
1122,They Made Me a Criminal (1939),
1123,Last Time I Saw Paris,
1124,Farewell to Arms,
1125,Innocents,
1126,Old Man and the Sea,
1127,Truman Show,1
1128,Heidi Fleiss: Hollywood Madam (1995),
1129,Chungking Express (1994),
1130,Jupiter's Wife (1994),
1131,Safe (1995),
1132,Feeling Minnesota (1996),
1133,Escape to Witch Mountain (1975),
1134,Get on the Bus (1996),
1135,Doors,
1136,Ghosts of Mississippi (1996),
1137,Beautiful Thing (1996),
1138,Best Men (1997),
1139,Hackers (1995),2
1140,Road to Wellville,
1141,War Room,
1142,When We Were Kings (1996),
1143,Hard Eight (1996),
1144,Quiet Room,
1145,Blue Chips (1994),
1146,Calendar Girl (1993),
1147,My Family (1995),
1148,Tom & Viv (1994),
1149,Walkabout (1971),
1150,Last Dance (1996),
1151,Original Gangstas (1996),
1152,In Love and War (1996),
1153,Backbeat (1993),
1154,Alphaville (1965),
1155,Rendezvous in Paris (Rendez-vous de Paris,
1156,Cyclo (1995),
1157,Relic,
1158,Fille seule,
1159,Stalker (1979),
1160,Love! Valour! Compassion! (1997),
1161,Palookaville (1996),
1162,Phat Beach (1996),
1163,Portrait of a Lady,
1164,Zeus and Roxanne (1997),
1165,Big Bully (1996),
1166,Love & Human Remains (1993),
1167,Sum of Us,
1168,Little Buddha (1993),
1169,Fresh (1994),
1170,Spanking the Monkey (1994),
1171,Wild Reeds (1994),
1172,Women,
1173,Bliss (1997),
1174,Caught (1996),
1175,Hugo Pool (1997),
1176,Welcome To Sarajevo (1997),
1177,Dunston Checks In (1996),
1178,Major Payne (1994),
1179,Man of the House (1995),
1180,I Love Trouble (1994),
1181,Low Down Dirty Shame,
1182,Cops and Robbersons (1994),
1183,Cowboy Way,
1184,Endless Summer 2,
1185,In the Army Now (1994),
1186,Inkwell,
1187,Switchblade Sisters (1975),
1188,Young Guns II (1990),
1189,Prefontaine (1997),
1190,That Old Feeling (1997),
1191,Letter From Death Row,
1192,Boys of St. Vincent,
1193,Before the Rain (Pred dozhdot) (1994),
1194,Once Were Warriors (1994),
1195,Strawberry and Chocolate (Fresa y chocolate) (1993),
1196,Savage Nights (Nuits fauves,
1197,Family Thing,
1198,Purple Noon (1960),
1199,Cemetery Man (Dellamorte Dellamore) (1994),
1200,Kim (1950),
1201,Marlene Dietrich: Shadow and Light (1996),
1202,Maybe,
1203,Top Hat (1935),
1204,To Be or Not to Be (1942),
1205,Secret Agent,
1206,Amos & Andrew (1993),
1207,Jade (1995),
1208,Kiss of Death (1995),
1209,Mixed Nuts (1994),
1210,Virtuosity (1995),
1211,Blue Sky (1994),
1212,Flesh and Bone (1993),
1213,Guilty as Sin (1993),
1214,In the Realm of the Senses (Ai no corrida) (1976),
1215,Barb Wire (1996),
1216,Kissed (1996),
1217,Assassins (1995),
1218,Friday (1995),
1219,Goofy Movie,
1220,Higher Learning (1995),
1221,When a Man Loves a Woman (1994),
1222,Judgment Night (1993),
1223,King of the Hill (1993),
1224,Scout,
1225,Angus (1995),
1226,Night Falls on Manhattan (1997),
1227,Awfully Big Adventure,
1228,Under Siege 2: Dark Territory (1995),
1229,Poison Ivy II (1995),
1230,Ready to Wear (Pret-A-Porter) (1994),
1231,Marked for Death (1990),
1232,Madonna: Truth or Dare (1991),
1233,Nénette et Boni (1996),
1234,Chairman of the Board (1998),
1235,Big Bang Theory,1
1236,Other Voices,
1237,Twisted (1996),
1238,Full Speed (1996),
1239,Cutthroat Island (1995),
1240,Ghost in the Shell (Kokaku kidotai) (1995),5
1241,Van,
1242,Old Lady Who Walked in the Sea,
1243,Night Flier (1997),
1244,Metro (1997),
1245,Gridlock'd (1997),
1246,Bushwhacked (1995),
1247,Bad Girls (1994),
1248,Blink (1994),
1249,For Love or Money (1993),
1250,Best of the Best 3: No Turning Back (1995),
1251,A Chef in Love (1996),
1252,Contempt (Mépris,
1253,Tie That Binds,
1254,Gone Fishin' (1997),
1255,Broken English (1996),
1256,Designated Mourner,
1257,Designated Mourner,
1258,Trial and Error (1997),
1259,Pie in the Sky (1995),
1260,Total Eclipse (1995),
1261,Run of the Country,
1262,Walking and Talking (1996),
1263,Foxfire (1996),
1264,Nothing to Lose (1994),
1265,Star Maps (1997),
1266,Bread and Chocolate (Pane e cioccolata) (1973),
1267,Clockers (1995),
1268,Bitter Moon (1992),
1269,Love in the Afternoon (1957),
1270,Life with Mikey (1993),
1271,North (1994),
1272,Talking About Sex (1994),
1273,Color of Night (1994),
1274,Robocop 3 (1993),
1275,Killer (Bulletproof Heart) (1994),
1276,Sunset Park (1996),
1277,Set It Off (1996),
1278,Selena (1997),
1279,Wild America (1997),
1280,Gang Related (1997),
1281,Manny & Lo (1996),
1282,Grass Harp,
1283,Out to Sea (1997),
1284,Before and After (1996),
1285,Princess Caraboo (1994),
1286,Shall We Dance? (1937),
1287,Ed (1996),
1288,Denise Calls Up (1995),
1289,Jack and Sarah (1995),
1290,Country Life (1994),
1291,Celtic Pride (1996),
1292,Simple Wish,
1293,Star Kid (1997),
1294,Ayn Rand: A Sense of Life (1997),
1295,Kicked in the Head (1997),
1296,Indian Summer (1996),
1297,Love Affair (1994),
1298,Band Wagon,
1299,Penny Serenade (1941),
1300,'Til There Was You (1997),
1301,Stripes (1981),
1302,Late Bloomers (1996),
1303,Getaway,
1304,New York Cop (1996),
1305,National Lampoon's Senior Trip (1995),
1306,Delta of Venus (1994),
1307,Carmen Miranda: Bananas Is My Business (1994),
1308,Babyfever (1994),
1309,Very Natural Thing,
1310,Walk in the Sun,
1311,Waiting to Exhale (1995),
1312,Pompatus of Love,
1313,Palmetto (1998),
1314,Surviving the Game (1994),
1315,Inventing the Abbotts (1997),
1316,Horse Whisperer,
1317,Journey of August King,
1318,Catwalk (1995),
1319,Neon Bible,
1320,Homage (1995),
1321,Open Season (1996),
1322,Metisse (Café au Lait) (1993),
1323,Wooden Man's Bride,
1324,Loaded (1994),
1325,August (1996),
1326,Boys (1996),
1327,Captives (1994),
1328,Of Love and Shadows (1994),
1329,Low Life,
1330,An Unforgettable Summer (1994),
1331,Last Klezmer: Leopold Kozlowski,
1332,My Life and Times With Antonin Artaud (En compagnie d'Antonin Artaud) (1993),
1333,Midnight Dancers (Sibak) (1994),
1334,Somebody to Love (1994),
1335,American Buffalo (1996),
1336,Kazaam (1996),
1337,Larger Than Life (1996),
1338,Two Deaths (1995),
1339,Stefano Quantestorie (1993),
1340,Crude Oasis,
1341,Hedd Wyn (1992),
1342,Convent,
1343,Lotto Land (1995),
1344,Story of Xinghua,
1345,Day the Sun Turned Cold,
1346,Dingo (1992),
1347,Ballad of Narayama,
1348,Every Other Weekend (1990),
1349,Mille bolle blu (1993),
1350,Crows and Sparrows (1949),
1351,Lover's Knot (1996),
1352,Shadow of Angels (Schatten der Engel) (1976),
1353,1-900 (1994),
1354,Venice/Venice (1992),
1355,Infinity (1996),
1356,Ed's Next Move (1996),
1357,For the Moment (1994),
1358,The Deadly Cure (1996),
1359,Boys in Venice (1996),
1360,Sexual Life of the Belgians,
1361,Search for One-eye Jimmy,
1362,American Strays (1996),
1363,Leopard Son,
1364,Bird of Prey (1996),
1365,Johnny 100 Pesos (1993),
1366,JLG/JLG - autoportrait de décembre (1994),
1367,Faust (1994),
1368,Mina Tannenbaum (1994),
1369,Forbidden Christ,
1370,I Can't Sleep (J'ai pas sommeil) (1994),
1371,Machine,
1372,Stranger,
1373,Good Morning (1971),
1374,Falling in Love Again (1980),
1375,Cement Garden,
1376,Meet Wally Sparks (1997),
1377,Hotel de Love (1996),
1378,Rhyme & Reason (1997),
1379,Love and Other Catastrophes (1996),
1380,Hollow Reed (1996),
1381,Losing Chase (1996),
1382,Bonheur,
1383,Second Jungle Book: Mowgli & Baloo,
1384,Squeeze (1996),
1385,Roseanna's Grave (For Roseanna) (1997),
1386,Tetsuo II: Body Hammer (1992),
1387,Fall (1997),
1388,Gabbeh (1996),
1389,Mondo (1996),
1390,Innocent Sleep,
1391,For Ever Mozart (1996),
1392,Locusts,
1393,Stag (1997),
1394,Swept from the Sea (1997),
1395,Hurricane Streets (1998),
1396,Stonewall (1995),
1397,Of Human Bondage (1934),
1398,Anna (1996),
1399,Stranger in the House (1997),
1400,Picture Bride (1995),
1401,M. Butterfly (1993),
1402,Ciao,
1403,Caro Diario (Dear Diary) (1994),
1404,Withnail and I (1987),
1405,Boy's Life 2 (1997),
1406,When Night Is Falling (1995),
1407,Specialist,
1408,Gordy (1995),
1409,Swan Princess,
1410,Harlem (1993),
1411,Barbarella (1968),
1412,Land Before Time III: The Time of the Great Giving (1995) (V),
1413,Street Fighter (1994),
1414,Coldblooded (1995),
1415,Next Karate Kid,
1416,No Escape (1994),
1417,Turning,
1418,Joy Luck Club,
1419,Highlander III: The Sorcerer (1994),
1420,Gilligan's Island: The Movie (1998),
1421,My Crazy Life (Mi vida loca) (1993),
1422,Suture (1993),
1423,Walking Dead,
1424,I Like It Like That (1994),
1425,I'll Do Anything (1994),
1426,Grace of My Heart (1996),
1427,Drunks (1995),
1428,SubUrbia (1997),
1429,Sliding Doors (1998),
1430,Ill Gotten Gains (1997),
1431,Legal Deceit (1997),
1432,Mighty,
1433,Men of Means (1998),
1434,Shooting Fish (1997),
1435,Steal Big,
1436,Mr. Jones (1993),
1437,House Party 3 (1994),
1438,Panther (1995),
1439,Jason's Lyric (1994),
1440,Above the Rim (1994),
1441,Moonlight and Valentino (1995),
1442,Scarlet Letter,
1443,8 Seconds (1994),
1444,That Darn Cat! (1965),
1445,Ladybird Ladybird (1994),
1446,Bye Bye,
1447,Century (1993),
1448,My Favorite Season (1993),
1449,Pather Panchali (1955),
1450,Golden Earrings (1947),
1451,Foreign Correspondent (1940),
1452,Lady of Burlesque (1943),
1453,Angel on My Shoulder (1946),
1454,Angel and the Badman (1947),
1455,Outlaw,
1456,Beat the Devil (1954),
1457,Love Is All There Is (1996),
1458,Damsel in Distress,
1459,Madame Butterfly (1995),
1460,Sleepover (1995),
1461,Here Comes Cookie (1935),
1462,Thieves (Voleurs,
1463,Boys,
1464,Stars Fell on Henrietta,
1465,Last Summer in the Hamptons (1995),
1466,Margaret's Museum (1995),
1467,Saint of Fort Washington,
1468,Cure,
1469,Tom and Huck (1995),
1470,Gumby: The Movie (1995),
1471,Hideaway (1995),
1472,Visitors,
1473,Little Princess,
1474,Nina Takes a Lover (1994),
1475,Bhaji on the Beach (1993),
1476,Raw Deal (1948),
1477,Nightwatch (1997),
1478,Dead Presidents (1995),
1479,Reckless (1995),
1480,Herbie Rides Again (1974),
1481,S.F.W. (1994),
1482,Gate of Heavenly Peace,
1483,Man in the Iron Mask,
1484,Jerky Boys,
1485,Colonel Chabert,
1486,Girl in the Cadillac (1995),
1487,Even Cowgirls Get the Blues (1993),
1488,Germinal (1993),
1489,Chasers (1994),
1490,Fausto (1993),
1491,Tough and Deadly (1995),
1492,Window to Paris (1994),
1493,Modern Affair,
1494,Mostro,
1495,Flirt (1995),
1496,Carpool (1996),
1497,Line King: Al Hirschfeld,
1498,Farmer & Chase (1995),
1499,Grosse Fatigue (1994),
1500,Santa with Muscles (1996),
1501,Prisoner of the Mountains (Kavkazsky Plennik) (1996),
1502,Naked in New York (1994),
1503,Gold Diggers: The Secret of Bear Mountain (1995),
1504,Bewegte Mann,
1505,Killer: A Journal of Murder (1995),
1506,Nelly & Monsieur Arnaud (1995),
1507,Three Lives and Only One Death (1996),
1508,Babysitter,
1509,Getting Even with Dad (1994),
1510,Mad Dog Time (1996),
1511,Children of the Revolution (1996),
1512,World of Apu,
1513,Sprung (1997),
1514,Dream With the Fishes (1997),
1515,Wings of Courage (1995),
1516,Wedding Gift,
1517,Race the Sun (1996),
1518,Losing Isaiah (1995),
1519,New Jersey Drive (1995),
1520,Fear,
1521,Mr. Wonderful (1993),
1522,Trial by Jury (1994),
1523,Good Man in Africa,
1524,Kaspar Hauser (1993),
1525,Object of My Affection,
1526,Witness (1985),
1527,Senseless (1998),
1528,Nowhere (1997),
1529,Underground (1995),
1530,Jefferson in Paris (1995),
1531,Far From Home: The Adventures of Yellow Dog (1995),
1532,Foreign Student (1994),
1533,I Don't Want to Talk About It (De eso no se habla) (1993),
1534,Twin Town (1997),
1535,Enfer,
1536,Aiqing wansui (1994),
1537,Cosi (1996),
1538,All Over Me (1997),
1539,Being Human (1993),
1540,Amazing Panda Adventure,
1541,Beans of Egypt,
1542,Scarlet Letter,
1543,Johns (1996),
1544,It Takes Two (1995),
1545,Frankie Starlight (1995),
1546,Shadows (Cienie) (1988),
1547,Show,
1548,The Courtyard (1995),
1549,Dream Man (1995),
1550,Destiny Turns on the Radio (1995),
1551,Glass Shield,
1552,Hunted,
1553,Underneath,
1554,Safe Passage (1994),
1555,Secret Adventures of Tom Thumb,
1556,Condition Red (1995),
1557,Yankee Zulu (1994),
1558,Aparajito (1956),
1559,Hostile Intentions (1994),
1560,Clean Slate (Coup de Torchon) (1981),
1561,Tigrero: A Film That Was Never Made (1994),
1562,Eye of Vichy,
1563,Promise,
1564,To Cross the Rubicon (1991),
1565,Daens (1992),
1566,Man from Down Under,
1567,Careful (1992),
1568,Vermont Is For Lovers (1992),
1569,Vie est belle,
1570,Quartier Mozart (1992),
1571,Touki Bouki (Journey of the Hyena) (1973),
1572,Wend Kuuni (God's Gift) (1982),
1573,Spirits of the Dead (Tre passi nel delirio) (1968),
1574,Pharaoh's Army (1995),
1575,I,
1576,Hungarian Fairy Tale,
1577,Death in the Garden (Mort en ce jardin,
1578,Collectionneuse,
1579,Baton Rouge (1988),
1580,Liebelei (1933),
1581,Woman in Question,
1582,T-Men (1947),
1583,Invitation,
1584,Symphonie pastorale,
1585,American Dream (1990),
1586,Lashou shentan (1992),
1587,Terror in a Texas Town (1958),
1588,Salut cousin! (1996),
1589,Schizopolis (1996),
1590,To Have,
1591,Duoluo tianshi (1995),
1592,Magic Hour,
1593,Death in Brunswick (1991),
1594,Everest (1998),
1595,Shopping (1994),
1596,Nemesis 2: Nebula (1995),
1597,Romper Stomper (1992),
1598,City of Industry (1997),
1599,Someone Else's America (1995),
1600,Guantanamera (1994),
1601,Office Killer (1997),
1602,Price Above Rubies,
1603,Angela (1995),
1604,He Walked by Night (1948),
1605,Love Serenade (1996),
1606,Deceiver (1997),
1607,Hurricane Streets (1998),
1608,Buddy (1997),
1609,B*A*P*S (1997),
1610,Truth or Consequences,
1611,Intimate Relations (1996),
1612,Leading Man,
1613,Tokyo Fist (1995),
1614,Reluctant Debutante,
1615,Warriors of Virtue (1997),
1616,Desert Winds (1995),
1617,Hugo Pool (1997),
1618,King of New York (1990),
1619,All Things Fair (1996),
1620,Sixth Man,
1621,Butterfly Kiss (1995),
1622,Paris,
1623,Cérémonie,
1624,Hush (1998),
1625,Nightwatch (1997),
1626,Nobody Loves Me (Keiner liebt mich) (1994),
1627,Wife,
1628,Lamerica (1994),
1629,Nico Icon (1995),
1630,Silence of the Palace,
1631,Slingshot,
1632,Land and Freedom (Tierra y libertad) (1995),
1633,Á köldum klaka (Cold Fever) (1994),
1634,Etz Hadomim Tafus (Under the Domin Tree) (1994),
1635,Two Friends (1986),
1636,Brothers in Trouble (1995),
1637,Girls Town (1996),
1638,Normal Life (1996),
1639,Bitter Sugar (Azucar Amargo) (1996),
1640,Eighth Day,
1641,Dadetown (1995),
1642,Some Mother's Son (1996),
1643,Angel Baby (1995),
1644,Sudden Manhattan (1996),
1645,Butcher Boy,
1646,Men With Guns (1997),
1647,Hana-bi (1997),
1648,Niagara,
1649,Big One,
1650,Butcher Boy,
1651,Spanish Prisoner,
1652,Temptress Moon (Feng Yue) (1996),
1653,Entertaining Angels: The Dorothy Day Story (1996),
1654,Chairman of the Board (1998),
1655,Favor,
1656,Little City (1998),
1657,Target (1995),
1658,Substance of Fire,
1659,Getting Away With Murder (1996),
1660,Small Faces (1995),
1661,New Age,
1662,Rough Magic (1995),
1663,Nothing Personal (1995),
1664,8 Heads in a Duffel Bag (1997),
1665,Brother's Kiss,
1666,Ripe (1996),
1667,Next Step,
1668,Wedding Bell Blues (1996),
1669,MURDER and murder (1996),
1670,Tainted (1998),
1671,Further Gesture,
1672,Kika (1993),
1673,Mirage (1995),
1674,Mamma Roma (1962),
1675,Sunchaser,
1676,War at Home,
1677,Sweet Nothing (1995),
1678,Mat' i syn (1997),
1679,B. Monkey (1998),
1680,Sliding Doors (1998),
1681,You So Crazy (1994),
1682,Scream of Stone (Schrei aus Stein) (1991),
This directory should hold execution results.
addSbtPlugin("com.typesafe.sbt" % "sbt-native-packager" % "1.7.4")
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.15.0")
#!/usr/bin/env bash
# If your default java install does not work, explicitly
# provide the path to the JDK 1.8 installation. On OSX
# with homebrew:
# export JAVA_HOME=/usr/local/Cellar/openjdk@8/1.8.0+282; ./run.sh
export JAVA_OPTS="-Xmx8G";
RUN=./logs/run-$(date "+%Y-%m-%d-%H:%M:%S")-$(hostname)
mkdir -p $RUN
LOGS=$RUN/log.txt
source ./config.sh
echo "------------------- OPTIMIZING ---------------------" >> $LOGS
sbt "runMain scaling.Optimizing --train $ML100Ku2base --test $ML100Ku2test --json $RUN/optimizing-100k.json --users 943 --movies 1682 --master local[1]" 2>&1 >>$LOGS
echo "------------------- DISTRIBUTED EXACT ---------------------" >> $LOGS
sbt "runMain distributed.Exact --train $ML100Ku2base --test $ML100Ku2test --json $RUN/exact-100k-4.json --k 10 --master local[4] --users 943 --movies 1682" 2>&1 >>$LOGS
sbt "runMain distributed.Exact --train $ML1Mrbtrain --test $ML1Mrbtest --separator :: --json $RUN/exact-1m-4.json --k 300 --master local[4] --users 6040 --movies 3952" 2>&1 >>$LOGS
echo "------------------- DISTRIBUTED APPROXIMATE ---------------------" >> $LOGS
sbt "runMain distributed.Approximate --train $ML100Ku2base --test $ML100Ku2test --json $RUN/approximate-100k-4-k10-r2.json --k 10 --master local[4] --users 943 --movies 1682 --partitions 10 --replication 2" 2>&1 >>$LOGS;
for R in 1 2 3 4 6 8; do
sbt "runMain distributed.Approximate --train $ML100Ku2base --test $ML100Ku2test --json $RUN/approximate-100k-4-k300-r$R.json --k 300 --master local[4] --users 943 --movies 1682 --partitions 10 --replication $R" 2>&1 >>$LOGS;
done
sbt "runMain distributed.Approximate --train $ML1Mrbtrain --test $ML1Mrbtest --separator :: --json $RUN/approximate-1m-4.json --k 300 --master local[4] --users 6040 --movies 3952 --partitions 8 --replication 1" 2>&1 >>$LOGS
echo "------------------- ECONOMICS -----------------------------------" >> $LOGS
sbt "runMain economics.Economics --json $RUN/economics.json" 2>&1 >>$LOGS
import org.rogach.scallop._
import org.apache.log4j.Logger
import org.apache.log4j.Level
import breeze.linalg._
import breeze.numerics._
import scala.io.Source
import scala.collection.mutable.ArrayBuffer
import ujson._
import org.apache.spark.rdd.RDD
import org.apache.spark.sql.SparkSession
import org.apache.log4j.Logger
import org.apache.log4j.Level
import shared.predictions._
package distributed {
class Conf(arguments: Seq[String]) extends ScallopConf(arguments) {
val train = opt[String](required = true)
val test = opt[String](required = true)
val k = opt[Int]()
val json = opt[String]()
val users = opt[Int]()
val movies = opt[Int]()
val separator = opt[String](default=Some("\t"))
val replication = opt[Int](default=Some(1))
val partitions = opt[Int](default=Some(1))
val master = opt[String]()
val num_measurements = opt[Int](default=Some(1))
verify()
}
object Approximate {
def main(args: Array[String]) {
var conf = new Conf(args)
// Remove these lines if encountering/debugging Spark
Logger.getLogger("org").setLevel(Level.OFF)
Logger.getLogger("akka").setLevel(Level.OFF)
val spark = conf.master.toOption match {
case None => SparkSession.builder().getOrCreate();
case Some(master) => SparkSession.builder().master(master).getOrCreate();
}
val sc = spark.sparkContext
println("")
println("******************************************************")
// conf object is not serializable, extract values that
// will be serialized with the parallelize implementations
val conf_users = conf.users()
val conf_movies = conf.movies()
val conf_k = conf.k()
println("Loading training data")
val train = loadSpark(sc, conf.train(), conf.separator(), conf.users(), conf.movies())
val test = loadSpark(sc, conf.test(), conf.separator(), conf.users(), conf.movies())
var knn : CSCMatrix[Double] = null
println("Partitioning users")
var partitionedUsers : Seq[Set[Int]] = partitionUsers(
conf.users(),
conf.partitions(),
conf.replication()
)
val measurements = (1 to scala.math.max(1,conf.num_measurements()))
.map(_ => timingInMs( () => {
// Use partitionedUsers here
0.0
}))
val mae = measurements(0)._1
val timings = measurements.map(_._2)
// Save answers as JSON
def printToFile(content: String,
location: String = "./answers.json") =
Some(new java.io.PrintWriter(location)).foreach{
f => try{
f.write(content)
} finally{ f.close }
}
conf.json.toOption match {
case None => ;
case Some(jsonFile) => {
val answers = ujson.Obj(
"Meta" -> ujson.Obj(
"train" -> ujson.Str(conf.train()),
"test" -> ujson.Str(conf.test()),
"k" -> ujson.Num(conf.k()),
"users" -> ujson.Num(conf.users()),
"movies" -> ujson.Num(conf.movies()),
"master" -> ujson.Str(sc.getConf.get("spark.master")),
"num-executors" -> ujson.Str(if (sc.getConf.contains("spark.executor.instances"))
sc.getConf.get("spark.executor.instances")
else
""),
"num_measurements" -> ujson.Num(conf.num_measurements()),
"partitions" -> ujson.Num(conf.partitions()),
"replication" -> ujson.Num(conf.replication())
),
"AK.1" -> ujson.Obj(
"knn_u1v1" -> ujson.Num(0.0),
"knn_u1v864" -> ujson.Num(0.0),
"knn_u1v344" -> ujson.Num(0.0),
"knn_u1v16" -> ujson.Num(0.0),
"knn_u1v334" -> ujson.Num(0.0),
"knn_u1v2" -> ujson.Num(0.0)
),
"AK.2" -> ujson.Obj(
"mae" -> ujson.Num(mae)
),
"AK.3" -> ujson.Obj(
"average (ms)" -> ujson.Num(mean(timings)),
"stddev (ms)" -> ujson.Num(std(timings))
)
)
val json = write(answers, 4)
println(json)
println("Saving answers in: " + jsonFile)
printToFile(json, jsonFile)
}
}
println("")
spark.stop()
}
}
}
import org.rogach.scallop._
import org.apache.log4j.Logger
import org.apache.log4j.Level
import breeze.linalg._
import breeze.numerics._
import scala.io.Source
import scala.collection.mutable.ArrayBuffer
import ujson._
import org.apache.spark.rdd.RDD
import org.apache.spark.sql.SparkSession
import org.apache.log4j.Logger
import org.apache.log4j.Level
import shared.predictions._
package distributed {
class ExactConf(arguments: Seq[String]) extends ScallopConf(arguments) {
val train = opt[String](required = true)
val test = opt[String](required = true)
val k = opt[Int](default=Some(10))
val json = opt[String]()
val users = opt[Int]()
val movies = opt[Int]()
val separator = opt[String](default=Some("\t"))
val master = opt[String]()
val num_measurements = opt[Int](default=Some(1))
verify()
}
object Exact {
def main(args: Array[String]) {
var conf = new ExactConf(args)
// Remove these lines if encountering/debugging Spark
Logger.getLogger("org").setLevel(Level.OFF)
Logger.getLogger("akka").setLevel(Level.OFF)
val spark = conf.master.toOption match {
case None => SparkSession.builder().getOrCreate();
case Some(master) => SparkSession.builder().master(master).getOrCreate();
}
spark.sparkContext.setLogLevel("ERROR")
val sc = spark.sparkContext
println("")
println("******************************************************")
// conf object is not serializable, extract values that
// will be serialized with the parallelize implementations
val conf_users = conf.users()
val conf_movies = conf.movies()
val conf_k = conf.k()
println("Loading training data from: " + conf.train())
val train = loadSpark(sc, conf.train(), conf.separator(), conf.users(), conf.movies())
val test = loadSpark(sc, conf.test(), conf.separator(), conf.users(), conf.movies())
val measurements = (1 to scala.math.max(1,conf.num_measurements())).map(_ => timingInMs( () => {
0.0
}))
val timings = measurements.map(_._2)
// Save answers as JSON
def printToFile(content: String,
location: String = "./answers.json") =
Some(new java.io.PrintWriter(location)).foreach{
f => try{
f.write(content)
} finally{ f.close }
}
conf.json.toOption match {
case None => ;
case Some(jsonFile) => {
val answers = ujson.Obj(
"Meta" -> ujson.Obj(
"train" -> ujson.Str(conf.train()),
"test" -> ujson.Str(conf.test()),
"k" -> ujson.Num(conf.k()),
"users" -> ujson.Num(conf.users()),
"movies" -> ujson.Num(conf.movies()),
"master" -> ujson.Str(sc.getConf.get("spark.master")),
"num-executors" -> ujson.Str(if (sc.getConf.contains("spark.executor.instances"))
sc.getConf.get("spark.executor.instances")
else
""),
"num_measurements" -> ujson.Num(conf.num_measurements())
),
"EK.1" -> ujson.Obj(
"1.knn_u1v1" -> ujson.Num(0.0),
"2.knn_u1v864" -> ujson.Num(0.0),
"3.knn_u1v886" -> ujson.Num(0.0),
"4.PredUser1Item1" -> ujson.Num(0.0),
"5.PredUser327Item2" -> ujson.Num(0.0),
"6.Mae" -> ujson.Num(0.0)
),
"EK.2" -> ujson.Obj(
"average (ms)" -> ujson.Num(mean(timings)), // Datatype of answer: Double
"stddev (ms)" -> ujson.Num(std(timings)) // Datatype of answer: Double
)
)
val json = write(answers, 4)
println(json)
println("Saving answers in: " + jsonFile)
printToFile(json, jsonFile)
}
}
println("")
spark.stop()
}
}
}
import org.rogach.scallop._
import breeze.linalg._
import breeze.numerics._
import scala.io.Source
import scala.collection.mutable.ArrayBuffer
import ujson._
package economics {
class Conf(arguments: Seq[String]) extends ScallopConf(arguments) {
val json = opt[String]()
verify()
}
object Economics {
def main(args: Array[String]) {
println("")
println("******************************************************")
var conf = new Conf(args)
// Save answers as JSON
def printToFile(content: String,
location: String = "./answers.json") =
Some(new java.io.PrintWriter(location)).foreach{
f => try{
f.write(content)
} finally{ f.close }
}
conf.json.toOption match {
case None => ;
case Some(jsonFile) => {
val answers = ujson.Obj(
"E.1" -> ujson.Obj(
"MinRentingDays" -> ujson.Num(0.0) // Datatype of answer: Double
),
"E.2" -> ujson.Obj(
"ContainerDailyCost" -> ujson.Num(0.0),
"4RPisDailyCostIdle" -> ujson.Num(0.0),
"4RPisDailyCostComputing" -> ujson.Num(0.0),
"MinRentingDaysIdleRPiPower" -> ujson.Num(0.0),
"MinRentingDaysComputingRPiPower" -> ujson.Num(0.0)
),
"E.3" -> ujson.Obj(
"NbRPisEqBuyingICCM7" -> ujson.Num(0.0),
"RatioRAMRPisVsICCM7" -> ujson.Num(0.0),
"RatioComputeRPisVsICCM7" -> ujson.Num(0.0)
)
)
val json = write(answers, 4)
println(json)
println("Saving answers in: " + jsonFile)
printToFile(json, jsonFile)
}
}
println("")
}
}
}
import org.rogach.scallop._
import breeze.linalg._
import breeze.numerics._
import scala.io.Source
import scala.collection.mutable.ArrayBuffer
import ujson._
import shared.predictions._
import org.apache.spark.sql.SparkSession
import org.apache.log4j.Logger
import org.apache.log4j.Level
package scaling {
class Conf(arguments: Seq[String]) extends ScallopConf(arguments) {
val train = opt[String](required = true)
val test = opt[String](required = true)
val json = opt[String]()
val users = opt[Int]()
val movies = opt[Int]()
val separator = opt[String](default=Some("\t"))
val master = opt[String]()
val num_measurements = opt[Int](default=Some(1))
verify()
}
object Optimizing extends App {
var conf = new Conf(args)
// conf object is not serializable, extract values that
// will be serialized with the parallelize implementations
val conf_users = conf.users()
val conf_movies = conf.movies()
// Remove these lines if encountering/debugging Spark
Logger.getLogger("org").setLevel(Level.OFF)
Logger.getLogger("akka").setLevel(Level.OFF)
val spark = conf.master.toOption match {
case None => SparkSession.builder().getOrCreate();
case Some(master) => SparkSession.builder().master(master).getOrCreate();
}
spark.sparkContext.setLogLevel("ERROR")
val sc = spark.sparkContext
println("Loading training data from: " + conf.train())
val train = loadSpark(sc, conf.train(), conf.separator(), conf.users(), conf.movies())
val test = loadSpark(sc, conf.test(), conf.separator(), conf.users(), conf.movies())
val measurements = (1 to conf.num_measurements()).map(x => timingInMs(() => {
0.0
}))
val timings = measurements.map(t => t._2)
val mae = measurements(0)._1
// Save answers as JSON
def printToFile(content: String,
location: String = "./answers.json") =
Some(new java.io.PrintWriter(location)).foreach{
f => try{
f.write(content)
} finally{ f.close }
}
conf.json.toOption match {
case None => ;
case Some(jsonFile) => {
val answers = ujson.Obj(
"Meta" -> ujson.Obj(
"train" -> ujson.Str(conf.train()),
"test" -> ujson.Str(conf.test()),
"users" -> ujson.Num(conf.users()),
"movies" -> ujson.Num(conf.movies()),
"master" -> ujson.Str(conf.master()),
"num_measurements" -> ujson.Num(conf.num_measurements())
),
"BR.1" -> ujson.Obj(
"1.k10u1v1" -> ujson.Num(0.0),
"2.k10u1v864" -> ujson.Num(0.0),
"3.k10u1v886" -> ujson.Num(0.0),
"4.PredUser1Item1" -> ujson.Num(0.0),
"5.PredUser327Item2" -> ujson.Num(0.0),
"6.Mae" -> ujson.Num(0.0)
),
"BR.2" -> ujson.Obj(
"average (ms)" -> ujson.Num(mean(timings)), // Datatype of answer: Double
"stddev (ms)" -> ujson.Num(std(timings)) // Datatype of answer: Double
)
)
val json = write(answers, 4)
println(json)
println("Saving answers in: " + jsonFile)
printToFile(json, jsonFile)
}
}
println("")
}
}
package predict
object Predictor extends App {
println("Computing predictions ...")
println("Done")
}
package recommend
object Recommender extends App {
println("Computing recommendations ...")
println("Done")
}
package shared
import breeze.linalg._
import breeze.numerics._
import scala.io.Source
import scala.collection.mutable.ArrayBuffer
import org.apache.spark.SparkContext
package object predictions
{
// ------------------------ For template
case class Rating(user: Int, item: Int, rating: Double)
def timingInMs(f : ()=>Double ) : (Double, Double) = {
val start = System.nanoTime()
val output = f()
val end = System.nanoTime()
return (output, (end-start)/1000000.0)
}
def toInt(s: String): Option[Int] = {
try {
Some(s.toInt)
} catch {
case e: Exception => None
}
}
def mean(s :Seq[Double]): Double = if (s.size > 0) s.reduce(_+_) / s.length else 0.0
def std(s :Seq[Double]): Double = {
if (s.size == 0) 0.0
else {
val m = mean(s)
scala.math.sqrt(s.map(x => scala.math.pow(m-x, 2)).sum / s.length.toDouble)
}
}
def load(path : String, sep : String, nbUsers : Int, nbMovies : Int) : CSCMatrix[Double] = {
val file = Source.fromFile(path)
val builder = new CSCMatrix.Builder[Double](rows=nbUsers, cols=nbMovies)
for (line <- file.getLines) {
val cols = line.split(sep).map(_.trim)
toInt(cols(0)) match {
case Some(_) => builder.add(cols(0).toInt-1, cols(1).toInt-1, cols(2).toDouble)
case None => None
}
}
file.close
builder.result()
}
def loadSpark(sc : org.apache.spark.SparkContext, path : String, sep : String, nbUsers : Int, nbMovies : Int) : CSCMatrix[Double] = {
val file = sc.textFile(path)
val ratings = file
.map(l => {
val cols = l.split(sep).map(_.trim)
toInt(cols(0)) match {
case Some(_) => Some(((cols(0).toInt-1, cols(1).toInt-1), cols(2).toDouble))
case None => None
}
})
.filter({ case Some(_) => true
case None => false })
.map({ case Some(x) => x
case None => ((-1, -1), -1) }).collect()
val builder = new CSCMatrix.Builder[Double](rows=nbUsers, cols=nbMovies)
for ((k,v) <- ratings) {
v match {
case d: Double => {
val u = k._1
val i = k._2
builder.add(u, i, d)
}
}
}
return builder.result
}
def partitionUsers (nbUsers : Int, nbPartitions : Int, replication : Int) : Seq[Set[Int]] = {
val r = new scala.util.Random(1337)
val bins : Map[Int, collection.mutable.ListBuffer[Int]] = (0 to (nbPartitions-1))
.map(p => (p -> collection.mutable.ListBuffer[Int]())).toMap
(0 to (nbUsers-1)).foreach(u => {
val assignedBins = r.shuffle(0 to (nbPartitions-1)).take(replication)
for (b <- assignedBins) {
bins(b) += u
}
})
bins.values.toSeq.map(_.toSet)
}
}
package test
import org.scalatest._
import funsuite._
import test.optimizing._
import test.distributed._
class AllTests extends Sequential(
new OptimizingTests,
new ExactTests,
new ApproximateTests
)
import org.scalatest.funsuite._
final class Prediction extends AnyFunSuite {
test("prediction works") {
assert(true)
}
}