2,665,933 events, 1,391,077 push events, 2,171,624 commit messages, 150,258,697 characters
Fuck yeah, I figured out that dumb ass insert logic.
Create README.md with Installation instructions
For brand new would-be julia users who want to get started with Julia, I would love to send them to this github repo to get started, but there's no information on this page explaining what to do once you're here!
I know that these tutorials were originally written with JuliaBox in mind, so i've tried to maintain that spirit in these instructions. But hopefully we can also include instructions here about how to get started with julia in any capacity, because JuliaBox may not be the preferred option for everyone, and these are great tutorials that it would be a shame to not share! :)
Likely section material pressure argue beyond role indeed. Recognize boy too really eat. Defense now oil word think. No church hard experience dog stop culture from.
"8:30am. I am up. Today I am going to do it properly. Let me chill just a bit and then I will start.
8:45am. Let me start. I can slack later.
I just have to level up a bit more. 2020 will be a year to just grind these skills. Once I set things up, I will be able to do AI properly in the sense of making continuous improvements to my agents.
RL is not special. Once things are done properly, it will go through the same way supervised learning did. I need to get to that point.
Damn, I really, really wish I could have been programming this in a functional language. TS is low level trash, even though it is far better than vanilla JS.
Just a little bit more and then I will be able to move on.
Let me get semantic-tokens-sample to work. I tried running it yesterday, but it failed. What I am going to do is get rid of the proposed API stuff and do it directly using the latest version of VS Code which has the feature integrated.
First thought let me study the code.
8:55am. Let me check out one thing - during the night I've been wondering why the KFAC update is squared. I had that written out in that draft paper, but redoing the thing in my head after all this time shows that it makes no sense.
c = a * b = 4, grad = 1
a = 1, b = 1 -> a' = 2, b = 1, c' = 2
a = 0.5, b = 2 -> a' = 2.5, b = 2, c' = 5
a = 0.25, b = 4 -> a' = 4.25, b = 4, c' = 17
What the hell is that c = a * b = 4. Clearly from the examples it should be 1.
c = a * b, grad = 1, a' = a + grad * b
a = 1, b = 1, c = 1 -> a' = 2, b = 1, c' = 2
a = 0.5, b = 2, c = 1 -> a' = 2.5, b = 2, c' = 5
a = 0.25, b = 4, c = 1 -> a' = 4.25, b = 4, c' = 17
9:05am.
///
Suppose b is a 1d whitening matrix, a is the weight matrix and c is the two combined. What is the update for different values of b? Considering that the update rule for a is a' = a + gradient * b then...
c = a * b, grad = 1, a' = a + grad * b
a = 1, b = 1, c = 1 -> a' = 2, b = 1, c' = 2
a = 0.5, b = 2, c = 1 -> a' = 2.5, b = 2, c' = 5
a = 0.25, b = 4, c = 1 -> a' = 4.25, b = 4, c' = 17
Now suppose a and b are not separate as in PRONG, but we have a * b and b like in KFAC. Then updating the weight matrix does not produce correct results using the c' = c + gradient * b rule.
c = a * b
c = 1 -> b = 1 -> c' = 2
c = 1 -> b = 2 -> c' = 3 // Should be 5.
c = 1 -> b = 4 -> c' = 5 // Should be 17.
As shown above the right relationship implies that right quantity for multiplying the gradient is the square of b. a * b after the update would be (a + gradient * b) * b = a * b + gradient * b * b. Meaning b * b is what should be used here to add to the gradient just as the first example implies.
The proof involving multidimensional matrices rather than scalars is similar.
///
Let me modify this a bit. I get it now. I completely forgot about this.
9:10am. Yeah, I see how b gets double counted. But this whole exercise shows that using scalars to transmit distance is not the correct way to go. Instead it should be frequency. If you want to make large moves, you need to repeat the update instead.
The mid point between backprop and reversibility would be some kind of frequency update scheme and it would have the best points of both. It might be more computationally expensive than either.
I've been thinking about these lines for a while, but this is the first time I've actually said it out loud. The way to go around instability issues in RL due to variance of rewards is to use frequency based updates.
I don't really know how to do this though...
9:15am. It would be tough to do in RNNs.
...There are various ways though.
Literally every I can think of would greatly be benefited by having weights on chips though. As I thought, to really get anywhere, I will have to ditch the GPUs.
9:20am. The way to derive frequency based updates from regular gradient updates is to imagine dividing the learning rate l by some positive value n and counter that by updating the net by n number of times.
Regular gradient updates are when n is 1. As n goes to infinity, you get complely smooth updates with no noise introduced by the optimization process.
By that I mean, it could be assumed that SGD updates make discontinouus jumps which introduces noise into the optimization process. This actually helps get it around the saddle points and local minima, but if the jumps are too large then the noise swamps the process and you get instability. As n goes to infinity both the benefit and disadvantage of noise goes away.
9:25am. Even ignoring the noise from large updates, you still need ensembles as deep nets are nonlinear. With nonlinearity, you get switching behavior internally and even a small update can lead to large policy moves. To fight that, it would be best to use ensembles.
9:35am. Yeah, this story is coherent. There are problems with making it work at scale, but it could be done.
I still do not understand though how spiking nets could work. What sort of update scheme would be good when the nets are discontinuous completely?
That will all have to be figured out.
I should be able to do it. This kind of engineering is right up my alley. I just need to set everything up.
Forget this for now. I need to get back to studying the sample code. That is my mission at this moment. The rest can wait for later."
Create index.js
A basic understanding of JavaScript is needed to follow this tutorial. Security is hard. Often when we build applications we want to allow only registered users to access the application. We want to be able to manage user accounts, see when they last logged in, be able to disable suspicious accounts and have a dashboard to view and manage all this data. We might also decide to support multi-factor authentication and social login.
But security isn’t just hard, it also takes a while to implement. What if there’s a service that could take away this part of the development hassle from you? Why spend weeks or months rolling your own auth? This is where Auth0 shines. In this tutorial, I’ll show you how to build a chat application with Pusher, add user authentication with Auth0 Lock, and manage users from the Auth0 dashboard.
Introduction to Auth0 Auth0 is an Authentication-as-a-Service (or Identity-as-a-Service) provider focused on encapsulating user authentication and management, which provides an SDK to allow developers to easily add authentication and manage users. Its user management dashboard allows for breach detection and multifactor authentication, and Passwordless login.
Building the application We will be building a chat application that’ll allow users to communicate with each other where everyone sees every other person’s messages. It’ll work similarly to how channels work in Slack: just one channel for everybody to communicate.
Here’s what we’ll be building:
Setting up the backend We’ll start by building the backend which will facilitate receiving and broadcasting chat messages, serving static files, and also setting up Auth0 and Pusher.
First, you’ll need to signup for a Pusher and Auth0 account. Go to pusher.com and auth0.com and sign up for an account. To use Pusher API we have to signup and create a Pusher app from the dashboard. We can create as many applications as we want and each one will get an application id and secret key which we’ll use to initialise a Pusher instance on client or server side code.
Create a new Pusher account To create a new Pusher app, click the Your apps side menu, then click the Create a new app button below the drawer. This brings up the setup wizard.
Enter a name for the application. In this case I’ll call it “chat”. Select a cluster. Select the option “Create app for multiple environments” if you want to have different instances for development, staging and production. Select Vanilla JS as the frontend and NodeJS as the backend. Complete the process by clicking Create App button to set up your app instance.
Since we’re building our backend in Node using Express, let’s initialise a new Node app and install the needed dependencies. Run the following command:
npm init and select the default options npm i --save body-parser express pusher to install express and the Pusher node package Add a new file called server.js which will contain logic to authenticate the Pusher client and also render the static files we’ll be adding later. This file will contain the content below:
var express = require('express');
var bodyParser = require('body-parser');
var Pusher = require('pusher');
var app = express();
app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: false }));
var pusher = new Pusher({ appId: APP_ID, key: APP_KEY, secret: APP_SECRET, cluster: eu });
app.post('/pusher/auth', function(req, res) {
var socketId = req.body.socket_id;
var channel = req.body.channel_name;
var auth = pusher.authenticate(socketId, channel);
res.send(auth);
});
app.post('/message', function(req, res) {
var message = req.body.message;
var name = req.body.name;
pusher.trigger( 'private-chat', 'message-added', { message, name });
res.sendStatus(200);
});
app.get('/',function(req,res){
res.sendFile('/public/index.html', {root: __dirname });
});
app.use(express.static(__dirname + '/public'));
var port = process.env.PORT || 5000;
app.listen(port, function () {
console.log(`app listening on port ${port}!`)
});
We instantiate Pusher by passing in an object that contains the details of our app ID and secret key, which can be found on the App Keys tab in your Pusher dashboard. Pusher also provides a mechanism for authenticating users to a channel at the point of subscription. To do this, we expose an endpoint on the server that will validate the request and respond with a success or failure. This endpoint will be called by Pusher client libraries and can be named anything. We used the default name for this endpoint on Pusher, which is /pusher/auth. The line var auth = pusher.authenticate(socketId, channel); authenticates the client with Pusher and returns an authentication code to the calling client.
To allow this file to run when we start npm, we update package.json with the following value:
"scripts": {
"start": "node server.js",
"test": "echo \"Error: no test specified\" && exit 1"
}
Create an Auth0 client To create an Auth0 client
Select Clients from the side menu. On the new page, click the Create Client button Enter a name for the app and select Single Page App as an option Click the Create button to create the client.
An Auth0 client provides us with Client Id and Secret which we’ll use to interact with Auth0 from the code. On the settings tab, we can see the Name, Client Id, Secret, Client Type and many more. I want to enable CORS for my domain http://localhost:5000, set the log out URL and the URL to redirect to after the user has been authenticated with Auth0. Update the following settings with http://localhost:5000
Allowed Callback URLs Allowed Logout URLs Allowed Origins (CORS) Building the frontend With the backend all good to go, we build the web page that will facilitate messaging. Create a folder named public which will contain the html and javascript file. Create two new files style.css and index.html with the following content:
style.css
@import url("http://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css");
.chat
{
list-style: none;
margin: 0;
padding: 0;
}
.chat li
{
margin-bottom: 10px;
padding-bottom: 5px;
border-bottom: 1px dotted #B3A9A9;
}
.chat li.left .chat-body
{
margin-left: 60px;
}
.chat li.right .chat-body
{
margin-right: 60px;
}
.chat li .chat-body p
{
margin: 0;
color: #777777;
}
.panel .slidedown .glyphicon, .chat .glyphicon
{
margin-right: 5px;
}
.body-panel
{
overflow-y: scroll;
height: 250px;
}
::-webkit-scrollbar-track
{
-webkit-box-shadow: inset 0 0 6px rgba(0,0,0,0.3);
background-color: #F5F5F5;
}
::-webkit-scrollbar
{
width: 12px;
background-color: #F5F5F5;
}
::-webkit-scrollbar-thumb
{
-webkit-box-shadow: inset 0 0 6px rgba(0,0,0,.3);
background-color: #555;
}
index.html
<!-- template from http://bootsnipp.com/snippets/6eWd -->
<!DOCTYPE html>
<html>
<head>
<!-- Latest compiled and minified CSS -->
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css" integrity="sha384-BVYiiSIFeK1dGmJRAkycuHAHRg32OmUcww7on3RYdg4Va+PmSTsz/K68vbdEjh4u" crossorigin="anonymous">
<!-- Optional theme -->
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap-theme.min.css" integrity="sha384-rHyoN1iRsVXV4nD0JutlnGaslCJuC7uwjduW9SVrLvRYooPp2bWYgmgJQIXwl/Sp" crossorigin="anonymous">
<script
src="https://code.jquery.com/jquery-2.2.4.min.js"
integrity="sha256-BbhdlvQf/xTY9gja0Dq3HiwQF8LaCRTXxZKRutelT44="
crossorigin="anonymous"></script>
<!-- Latest compiled and minified JavaScript -->
<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js" integrity="sha384-Tc5IQib027qvyjSMfHjOMaLkfuWVxZxUPnCJA7l2mCWNIpG9mGCD8wGNIcPD7Txa" crossorigin="anonymous"></script>
<link rel="stylesheet" href="style.css">
<script src="https://cdn.auth0.com/js/lock/10.18.0/lock.min.js"></script>
<script src="https://js.pusher.com/4.0/pusher.min.js"></script>
<script src="index.js"></script>
</head>
<body>
<div class="container">
<div class="row form-group">
<div class="col-xs-12 col-md-offset-2 col-md-8 col-lg-8 col-lg-offset-2">
<div class="panel panel-primary">
<div class="panel-heading">
<span class="glyphicon glyphicon-comment"></span> <span id="username"></span>
<div class="btn-group pull-right">
<button type="button" class="btn btn-default btn-xs dropdown-toggle" data-toggle="dropdown">
<span class="glyphicon glyphicon-chevron-down"></span>
</button>
<ul class="dropdown-menu slidedown">
<li><a><span class="glyphicon glyphicon-refresh">
</span>Refresh</a></li>
<li><a><span class="glyphicon glyphicon-ok-sign">
</span>Available</a></li>
<li><a><span class="glyphicon glyphicon-remove">
</span>Busy</a></li>
<li><a><span class="glyphicon glyphicon-time"></span>
Away</a></li>
<li class="divider"></li>
<li><a id="logout"><span class="glyphicon glyphicon-off"></span>
Sign Out</a></li>
</ul>
</div>
</div>
<div class="panel-body body-panel">
<ul class="chat">
</ul>
</div>
<div class="panel-footer clearfix">
<textarea id="message" class="form-control" rows="3"></textarea>
<span class="col-lg-6 col-lg-offset-3 col-md-6 col-md-offset-3 col-xs-12" style="margin-top: 10px">
<button class="btn btn-warning btn-lg btn-block" id="btn-chat">Send</button>
</span>
</div>
</div>
</div>
</div>
</div>
<script id="new-message" type="text/template">
<li id="{{id}}" class="right clearfix">
<div class="chat-body clearfix">
<div class="header">
<small class="text-muted">{{name}}</small>
</div>
<p>
{{body}}
</p>
</div>
</li>
</script>
</body>
</html>
This file uses template from bootsnip and also includes a script reference to Auth0 Lock <script src="https://cdn.auth0.com/js/lock/10.18.0/lock.min.js"></script>. Lock is a drop-in authentication widget that provides a standard set of behaviours required for and a customisable user interface. It provides a simple way to integrate with Auth0 with very minimal configuration.
We want to allow users to sign in when they enter the application and be able to send messages once they’re authenticated. Add a new file index.js with the following content:
$(document).ready(function(){
// Initiating our Auth0Lock
let lock = new Auth0Lock(
'CLIENT_ID',
'CLIENT_DOMAIN',//example: lotus.auth0.com
{
auth: {
params: {
scope: 'openid profile'
}
},
autoclose: true,
closable: false,
rememberLastLogin: true
}
);
let profile = JSON.parse(localStorage.getItem('profile'));
let isAuthenticated = localStorage.getItem('isAuthenticated');
function updateValues(userProfile, authStatus) {
profile = userProfile;
isAuthenticated = authStatus;
}
if(!isAuthenticated && !window.location.hash){
lock.show();//show Lock widget
}
// Listening for the authenticated event
lock.on("authenticated", function(authResult) {
// Use the token in authResult to getUserInfo() and save it to localStorage
lock.getUserInfo(authResult.accessToken, function(error, profile) {
if (error) {
// Handle error
return;
}
localStorage.setItem('accessToken', authResult.accessToken);
localStorage.setItem('profile', JSON.stringify(profile));
localStorage.setItem('isAuthenticated', true);
updateValues(profile, true);
$("#username").html(profile.name);
});
});
});
We initialise Lock by passing it the Client Id of the app, your user domain which starts with your username followed by .auth0.com or .{YOUR_SELECTED_REGION}.auth0.com e.g lotus.eu.auth0.com. The widget is configurable and we can send in configuration options like closable, autoClose, and auth. Within the auth option we tell it to return the openid and profile claims.
We check if the user is authenticated and show the widget when they’re not. Once the user is authenticated, Lock emits the authenticated event which we’ve subscribed to. When it’s raised, we store the user profile and other credentials to localStorage and set the user’s name to be displayed on the page. Once the user is authenticated, we want to connect to Pusher and send messages across. Update index.js with the following code:
kthread: add kthread_create_worker*()
Kthread workers are currently created using the classic kthread API, namely kthread_run(). kthread_worker_fn() is passed as the @threadfn parameter.
This patch defines kthread_create_worker() and kthread_create_worker_on_cpu() functions that hide implementation details.
They enforce using kthread_worker_fn() for the main thread. But I doubt that there are any plans to create any alternative. In fact, I think that we do not want any alternative main thread because it would be hard to support consistency with the rest of the kthread worker API.
The naming and function of kthread_create_worker() is inspired by the workqueues API like the rest of the kthread worker API.
The kthread_create_worker_on_cpu() variant is motivated by the original kthread_create_on_cpu(). Note that we need to bind per-CPU kthread workers already when they are created. It makes the life easier. kthread_bind() could not be used later for an already running worker.
This patch does not convert existing kthread workers. The kthread worker API need more improvements first, e.g. a function to destroy the worker.
IMPORTANT:
kthread_create_worker_on_cpu() allows to use any format of the worker name, in compare with kthread_create_on_cpu(). The good thing is that it is more generic. The bad thing is that most users will need to pass the cpu number in two parameters, e.g. kthread_create_worker_on_cpu(cpu, "helper/%d", cpu).
To be honest, the main motivation was to avoid the need for an empty va_list. The only legal way was to create a helper function that would be called with an empty list. Other attempts caused compilation warnings or even errors on different architectures.
There were also other alternatives, for example, using #define or splitting __kthread_create_worker(). The used solution looked like the least ugly.
Link: http://lkml.kernel.org/r/1470754545-17632-6-git-send-email-pmladek@suse.com Signed-off-by: Petr Mladek pmladek@suse.com Acked-by: Tejun Heo tj@kernel.org Cc: Oleg Nesterov oleg@redhat.com Cc: Ingo Molnar mingo@redhat.com Cc: Peter Zijlstra peterz@infradead.org Cc: Steven Rostedt rostedt@goodmis.org Cc: "Paul E. McKenney" paulmck@linux.vnet.ibm.com Cc: Josh Triplett josh@joshtriplett.org Cc: Thomas Gleixner tglx@linutronix.de Cc: Jiri Kosina jkosina@suse.cz Cc: Borislav Petkov bp@suse.de Cc: Michal Hocko mhocko@suse.cz Cc: Vlastimil Babka vbabka@suse.cz Signed-off-by: Andrew Morton akpm@linux-foundation.org Signed-off-by: Linus Torvalds torvalds@linux-foundation.org Signed-off-by: celtare21 celtare21@gmail.com
Create NoSQL_Data_Models
When Not to Use SQL: Need high Availability in the data: Indicates the system is always up and there is no downtime Have Large Amounts of Data Need Linear Scalability: The need to add more nodes to the system so performance will increase linearly Low Latency: Shorter delay before the data is transferred once the instruction for the transfer has been received. Need fast reads and write Here is a helpful blog that describes the different types of NoSQL databases. You can bookmark it to review this later too.
Eventual Consistency: Over time (if no new changes are made) each copy of the data will be the same, but if there are new changes, the data may be different in different locations. The data may be inconsistent for only milliseconds. There are workarounds in place to prevent getting stale data.
Commonly Asked Questions: What does the network look like? Can you share any examples? In Apache Cassandra every node is connected to every node -- it's peer to peer database architecture.
Is data deployment strategy an important element of data modeling in Apache Cassandra? Deployment strategies are a great topic, but have very little to do with data modeling. Developing deployment strategies focuses on determining how many clusters to create or determining how many nodes are needed. These are topics generally covered under database architecture, database deployment and operations, which we will not cover in this lesson. Here is a useful link to learn more about it for Apache Cassandra.
In general, the size of your data and your data model can affect your deployment strategies. You need to think about how to create a cluster, how many nodes should be in that cluster, how to do the actual installation. More information about deployment strategies can be found on this DataStax documentation page
Citation for above slides: Here is the Wikipedia page cited in the slides.
Cassandra Architecture We are not going into a lot of details about the Apache Cassandra Architecture. However, if you would like to learn more about it for your job, here are some links that you may find useful.
Apache Cassandra Data Architecture:
Understanding the architecture Cassandra Architecture The following link will go more in-depth about the Apache Cassandra Data Model, how Cassandra reads, writes, updates, and deletes data.
Cassandra Documentation
CAP Theorem: Consistency: Every read from the database gets the latest (and correct) piece of data or an error
Availability: Every request is received and a response is given -- without a guarantee that the data is the latest update
Partition Tolerance: The system continues to work regardless of losing network connectivity between nodes
Additional Resource: You can also check out this Wikipedia page on the CAP theorem.
Commonly Asked Questions: Is Eventual Consistency the opposite of what is promised by SQL database per the ACID principle? Much has been written about how Consistency is interpreted in the ACID principle and the CAP theorem. Consistency in the ACID principle refers to the requirement that only transactions that abide by constraints and database rules are written into the database, otherwise the database keeps previous state. In other words, the data should be correct across all rows and tables. However, consistency in the CAP theorem refers to every read from the database getting the latest piece of data or an error. To learn more, you may find this discussion useful:
Discussion about ACID vs. CAP Which of these combinations is desirable for a production system - Consistency and Availability, Consistency and Partition Tolerance, or Availability and Partition Tolerance? As the CAP Theorem Wikipedia entry says, "The CAP theorem implies that in the presence of a network partition, one has to choose between consistency and availability." So there is no such thing as Consistency and Availability in a distributed database since it must always tolerate network issues. You can only have Consistency and Partition Tolerance (CP) or Availability and Partition Tolerance (AP). Remember, relational and non-relational databases do different things, and that's why most companies have both types of database systems.
Does Cassandra meet just Availability and Partition Tolerance in the CAP theorem? According to the CAP theorem, a database can actually only guarantee two out of the three in CAP. So supporting Availability and Partition Tolerance makes sense, since Availability and Partition Tolerance are the biggest requirements.
If Apache Cassandra is not built for consistency, won't the analytics pipeline break? If I am trying to do analysis, such as determining a trend over time, e.g., how many friends does John have on Twitter, and if you have one less person counted because of "eventual consistency" (the data may not be up-to-date in all locations), that's OK. In theory, that can be an issue but only if you are not constantly updating. If the pipeline pulls data from one node and it has not been updated, then you won't get it. Remember, in Apache Cassandra it is about Eventual Consistency.
Data Modeling in Apache Cassandra: Denormalization is not just okay -- it's a must Denormalization must be done for fast reads Apache Cassandra has been optimized for fast writes ALWAYS think Queries first One table per query is a great strategy Apache Cassandra does not allow for JOINs between tables Commonly Asked Questions: I see certain downsides of this approach, since in a production application, requirements change quickly and I may need to improve my queries later. Isn't that a downside of Apache Cassandra? In Apache Cassandra, you want to model your data to your queries, and if your business need calls for quickly changing requirements, you need to create a new table to process the data. That is a requirement of Apache Cassandra. If your business needs calls for ad-hoc queries, these are not a strength of Apache Cassandra. However keep in mind that it is easy to create a new table that will fit your new query. Additional Resource: Here is a reference to the DataStax documents on [Apache Cassandra].(https://docs.datastax.com/en/dse/6.7/cql/cql/ddl/dataModelingApproach.html)
Primary Key Must be unique The PRIMARY KEY is made up of either just the PARTITION KEY or may also include additional CLUSTERING COLUMNS A Simple PRIMARY KEY is just one column that is also the PARTITION KEY. A Composite PRIMARY KEY is made up of more than one column and will assist in creating a unique value and in your retrieval queries The PARTITION KEY will determine the distribution of data across the system Here is the DataStax documentation on Primary Keys.
Clustering Columns: The clustering column will sort the data in sorted ascending order, e.g., alphabetical order. Note: this is a mistake in the video, which says descending order. More than one clustering column can be added (or none!) From there the clustering columns will sort in order of how they were added to the primary key Commonly Asked Questions: How many clustering columns can we add? You can use as many clustering columns as you would like. You cannot use the clustering columns out of order in the SELECT statement. You may choose to omit using a clustering column in your SELECT statement. That's OK. Just remember to use them in order when you are using the SELECT statement.
Additional Resources: Here is the DataStax documentation on Composite Partition Keys. This Stackoverflow page provides a nice description of the difference between Partition Keys and Clustering Keys.
WHERE clause Data Modeling in Apache Cassandra is query focused, and that focus needs to be on the WHERE clause Failure to include a WHERE clause will result in an error Additional Resource AVOID using "ALLOW FILTERING": Here is a reference in DataStax that explains ALLOW FILTERING and why you should not use it.
Commonly Asked Questions: Why do we need to use a WHERE statement since we are not concerned about analytics? Is it only for debugging purposes? The WHERE statement is allowing us to do the fast reads. With Apache Cassandra, we are talking about big data -- think terabytes of data -- so we are making it fast for read purposes. Data is spread across all the nodes. By using the WHERE statement, we know which node to go to, from which node to get that data and serve it back. For example, imagine we have 10 years of data on 10 nodes or servers. So 1 year's data is on a separate node. By using the WHERE year = 1 statement we know which node to visit fast to pull the data from.
link_path_walk: nd->depth massage, part 1
nd->stack[0] is unused until the handling of trailing symlinks and we want to get rid of that. Having fucked that transformation up several times, I went for bloody pedantic series of provably equivalent transformations. Sorry.
Step 1: keep nd->depth higher by one in link_path_walk() - increment upon entry, decrement on exits, adjust the arithmetics inside and surround the calls of functions that care about nd->depth value (nd_alloc_stack(), get_link(), put_link()) with decrement/increment pairs.
Signed-off-by: Al Viro viro@zeniv.linux.org.uk
nsproxy: attach to namespaces via pidfds
For quite a while we have been thinking about using pidfds to attach to namespaces. This patchset has existed for about a year already but we've wanted to wait to see how the general api would be received and adopted. Now that more and more programs in userspace have started using pidfds for process management it's time to send this one out.
This patch makes it possible to use pidfds to attach to the namespaces of another process, i.e. they can be passed as the first argument to the setns() syscall. When only a single namespace type is specified the semantics are equivalent to passing an nsfd. That means setns(nsfd, CLONE_NEWNET) equals setns(pidfd, CLONE_NEWNET). However, when a pidfd is passed, multiple namespace flags can be specified in the second setns() argument and setns() will attach the caller to all the specified namespaces all at once or to none of them. Specifying 0 is not valid together with a pidfd.
Here are just two obvious examples: setns(pidfd, CLONE_NEWPID | CLONE_NEWNS | CLONE_NEWNET); setns(pidfd, CLONE_NEWUSER); Allowing to also attach subsets of namespaces supports various use-cases where callers setns to a subset of namespaces to retain privilege, perform an action and then re-attach another subset of namespaces.
If the need arises, as Eric suggested, we can extend this patchset to assume even more context than just attaching all namespaces. His suggestion specifically was about assuming the process' root directory when setns(pidfd, 0) or setns(pidfd, SETNS_PIDFD) is specified. For now, just keep it flexible in terms of supporting subsets of namespaces but let's wait until we have users asking for even more context to be assumed. At that point we can add an extension.
The obvious example where this is useful is a standard container manager interacting with a running container: pushing and pulling files or directories, injecting mounts, attaching/execing any kind of process, managing network devices all these operations require attaching to all or at least multiple namespaces at the same time. Given that nowadays most containers are spawned with all namespaces enabled we're currently looking at at least 14 syscalls, 7 to open the /proc//ns/ nsfds, another 7 to actually perform the namespace switch. With time namespaces we're looking at about 16 syscalls. (We could amortize the first 7 or 8 syscalls for opening the nsfds by stashing them in each container's monitor process but that would mean we need to send around those file descriptors through unix sockets everytime we want to interact with the container or keep on-disk state. Even in scenarios where a caller wants to join a particular namespace in a particular order callers still profit from batching other namespaces. That mostly applies to the user namespace but all container runtimes I found join the user namespace first no matter if it privileges or deprivileges the container similar to how unshare behaves.) With pidfds this becomes a single syscall no matter how many namespaces are supposed to be attached to.
A decently designed, large-scale container manager usually isn't the parent of any of the containers it spawns so the containers don't die when it crashes or needs to update or reinitialize. This means that for the manager to interact with containers through pids is inherently racy especially on systems where the maximum pid number is not significicantly bumped. This is even more problematic since we often spawn and manage thousands or ten-thousands of containers. Interacting with a container through a pid thus can become risky quite quickly. Especially since we allow for an administrator to enable advanced features such as syscall interception where we're performing syscalls in lieu of the container. In all of those cases we use pidfds if they are available and we pass them around as stable references. Using them to setns() to the target process' namespaces is as reliable as using nsfds. Either the target process is already dead and we get ESRCH or we manage to attach to its namespaces but we can't accidently attach to another process' namespaces. So pidfds lend themselves to be used with this api. The other main advantage is that with this change the pidfd becomes the only relevant token for most container interactions and it's the only token we need to create and send around.
Apart from significiantly reducing the number of syscalls from double digit to single digit which is a decent reason post-spectre/meltdown this also allows to switch to a set of namespaces atomically, i.e. either attaching to all the specified namespaces succeeds or we fail. If we fail we haven't changed a single namespace. There are currently three namespaces that can fail (other than for ENOMEM which really is not very interesting since we then have other problems anyway) for non-trivial reasons, user, mount, and pid namespaces. We can fail to attach to a pid namespace if it is not our current active pid namespace or a descendant of it. We can fail to attach to a user namespace because we are multi-threaded or because our current mount namespace shares filesystem state with other tasks, or because we're trying to setns() to the same user namespace, i.e. the target task has the same user namespace as we do. We can fail to attach to a mount namespace because it shares filesystem state with other tasks or because we fail to lookup the new root for the new mount namespace. In most non-pathological scenarios these issues can be somewhat mitigated. But there are cases where we're half-attached to some namespace and failing to attach to another one. I've talked about some of these problem during the hallway track (something only the pre-COVID-19 generation will remember) of Plumbers in Los Angeles in 2018(?). Even if all these issues could be avoided with super careful userspace coding it would be nicer to have this done in-kernel. Pidfds seem to lend themselves nicely for this.
The other neat thing about this is that setns() becomes an actual counterpart to the namespace bits of unshare().
Cc: Eric W. Biederman ebiederm@xmission.com Cc: Serge Hallyn serge@hallyn.com Cc: Jann Horn jannh@google.com Cc: Michael Kerrisk mtk.manpages@gmail.com Cc: Aleksa Sarai cyphar@cyphar.com Signed-off-by: Christian Brauner christian.brauner@ubuntu.com
If we agree that this is useful than I'd pick this up for for v5.8. There's probably some smart trick around nsproxy and pidns life-cycle management that I'm missing but I tried to be conservative wrt to taking references. /* v2 */
- Michael Kerrisk mtk.manpages@gmail.com:
- Michael pointed out that the semantics for setns(nsfd, 0) and setns(pidfd, 0) are not comparable. setns(pidfd, 0) is now disallowed completely. Users wanting to attach to all namespaces should simply specify them explicitly just as with unshare() and clone3().
- Jann Horn jannh@google.com:
- Jann pointed out that the setns() in its first form wasn't atomic in that userspace could end up in an intermediate state where e.g. the process had moved into the target user namespace but failed to move into the target mount namespace. In this new version I've removed all intermediate states. There's an installation/preparation state and a commit state similar to prepare_creds() and commit_creds().
/* v3 */
- Christian Brauner christian.brauner@ubuntu.com:
- The patchset is mostly unchanged. It was only fixed-up in response to changes in earlier patches.
/* v4 */ unchanged
[Port] Citadel Atmos Adjustments and Relief Valves (#156)
-
Oh god oh fuck I touched Atmos code
-
The more I use tgui the more I hate it
Shortens lines, removes "ref", adds AnimatedNumbers include
- Fucking DME keeps changing now matter how much I fix it
Co-authored-by: Yodeler superjacob50@yahoo.com Co-authored-by: Superyodeler 51685082+Superyodeler@users.noreply.github.com
Multiple
- Added main.lua
- Changed License from GPLv3 to Unlicense because fuck you
- Modified README
"1:50pm. Done with breakfast. Let me chill for a while, do the chores and then I'll move to the next example.
3:15pm. That break was a bit too long, but that is fine. If my sessions are too long like 2 days ago, I quickly run out of steam and need to recharge more.
It is best to pace yourself after all.
Let me start. Time for the snippet-sample.
{
"Print to console": {
"prefix": "log",
"body": [
"console.log('$1');",
"$2"
],
"description": "Log output to console"
}
}
Apart from the stuff in package.json, this is the entirety of it. What does $2 in the body do?
...Well, I am not sure what it does, but that second string does add an extra empty line.
Ok, done with that sample. Let me move to the next.
The next is source-control-sample.
3:30pm. This one is complex. Even the readme itself is 150 lines long.
3:45pm. Since it is a source control plugin, I think I will just skip it. Since it is close to 1k LOC refactoring all of it would too much of a drain on my time. Though something like this does seem like it would benefit from getting the reactive extensions treatment.
...Well, regardless, I'll get plenty of practice doing my own language plugin so there is not need to dawdle.
Let me move on to statusbar-sample.
3:55pm.
/*---------------------------------------------------------
* Copyright (C) Microsoft Corporation. All rights reserved.
*--------------------------------------------------------*/
import * as vscode from 'vscode';
let myStatusBarItem: vscode.StatusBarItem;
export function activate({ subscriptions }: vscode.ExtensionContext) {
// register a command that is invoked when the status bar
// item is selected
const myCommandId = 'sample.showSelectionCount';
subscriptions.push(vscode.commands.registerCommand(myCommandId, () => {
let n = getNumberOfSelectedLines(vscode.window.activeTextEditor);
vscode.window.showInformationMessage(`Yeah, ${n} line(s) selected... Keep going!`);
}));
// create a new status bar item that we can now manage
myStatusBarItem = vscode.window.createStatusBarItem(vscode.StatusBarAlignment.Right, 100);
myStatusBarItem.command = myCommandId;
subscriptions.push(myStatusBarItem);
// register some listener that make sure the status bar
// item always up-to-date
subscriptions.push(vscode.window.onDidChangeActiveTextEditor(updateStatusBarItem));
subscriptions.push(vscode.window.onDidChangeTextEditorSelection(updateStatusBarItem));
// update status bar item once at start
updateStatusBarItem();
}
function updateStatusBarItem(): void {
let n = getNumberOfSelectedLines(vscode.window.activeTextEditor);
if (n > 0) {
myStatusBarItem.text = `$(megaphone) ${n} line(s) selected`;
myStatusBarItem.show();
} else {
myStatusBarItem.hide();
}
}
function getNumberOfSelectedLines(editor: vscode.TextEditor | undefined): number {
let lines = 0;
if (editor) {
lines = editor.selections.reduce((prev, curr) => prev + (curr.end.line - curr.start.line), 0);
}
return lines;
}
Somehow I already remember refactoring this once before.
Well, let me just do it. The stuff in here is just such poor style.
4pm.
const getNumberOfSelectedLines = (editor: vscode.TextEditor | undefined): number => {
let lines = 0;
if (editor) {
lines = editor.selections.reduce((prev, curr) => prev + (curr.end.line - curr.start.line), 0);
}
return lines;
}
...This code is really such crap style.
const getNumberOfSelectedLines = (editor: vscode.TextEditor | undefined): number =>
editor ? editor.selections.reduce((prev, curr) => prev + (curr.end.line - curr.start.line), 0) : 0;
Here is a better way of doing it.
4:05pm.
/*---------------------------------------------------------
* Copyright (C) Microsoft Corporation. All rights reserved.
*--------------------------------------------------------*/
import * as vscode from 'vscode';
export function activate({ subscriptions }: vscode.ExtensionContext) {
const getNumberOfSelectedLines = (editor: vscode.TextEditor | undefined): number =>
editor ? editor.selections.reduce((prev, curr) => prev + (curr.end.line - curr.start.line), 0) : 0;
// register a command that is invoked when the status bar
// item is selected
const myCommandId = 'sample.showSelectionCount';
subscriptions.push(vscode.commands.registerCommand(myCommandId, () => {
const n = getNumberOfSelectedLines(vscode.window.activeTextEditor);
vscode.window.showInformationMessage(`Yeah, ${n} line(s) selected... Keep going!`);
}));
// create a new status bar item that we can now manage
const myStatusBarItem = vscode.window.createStatusBarItem(vscode.StatusBarAlignment.Right, 1000);
myStatusBarItem.command = myCommandId;
subscriptions.push(myStatusBarItem);
const updateStatusBarItem = (): void => {
const n = getNumberOfSelectedLines(vscode.window.activeTextEditor);
if (n > 0) {
myStatusBarItem.text = `$(megaphone) ${n} line(s) selected`;
myStatusBarItem.show();
} else {
myStatusBarItem.hide();
}
};
// register some listener that make sure the status bar
// item always up-to-date
subscriptions.push(
vscode.window.onDidChangeActiveTextEditor(updateStatusBarItem),
vscode.window.onDidChangeTextEditorSelection(updateStatusBarItem));
// update status bar item once at start
updateStatusBarItem();
}
Here it is. I remember fiddling with this months ago.
Let me run it and I'll move to the next thing.
4:10pm. Let me move on. This one is simple.
No way will I be able to keep all this in mind after I am done. I'll have to review for a bit.
The next one is task-provider-sample. Including this one, that is 8 samples left. I am nearly done.
4:20pm. Yeah, now that I've skipped one, I am in the skip everything stage. I really don't feel like redesigning that Rake task thing.
4:25pm. Stop surfing /a/. Let me just finish this. If needed I'll end this adventure today so I can move to working on Spiral tomorrow.
Skipping terminal-sample as well. Next, next, next!
theme-sample is beyond my interest. It is not a plugin. Next!
4:35pm. Actually, the stuff in tree-view-sample is of interest to me. It has some of the things I will need. Something like a file explorer is what I will need to do a module explorer for Spiral.
Forget that for now. Let me just move on. I've had a change of mind. I'll finish this all today. I am done with these samples. I've been doing them for two weeks and I think it is time I start wrapping up. It is not like I have to do all of them. If I need anything more past this, I'll do it on the fly as I go along. It is time to do some reviewing after I finish this. I want to display my work to those guys, so I'll open an issue for this.
4:55pm. I am done. Finito. Finally. Let me make a fork of the samples repo and I will push this there.
7:10pm.
///
A fork of the original repo. As an exercise, I did a significant amount of refactoring in the first 30 samples or so. Check out all the repos with extension'.ts files. Some of them have minor changes, but for others a significant effort from my own end went into refactoring them. A lot of the samples had broken packages so I fixed that along the way.
I've improved the readability in everything that I've touched, but if I were to pick some highlights here they are:
-
quickinput-sample- The author went hog wild with Promises in themultiStepInput.tsexample. Using reactive extensions, I rewrote this so it uses the MVU pattern, and cut the size of the code down by almost a half while significantly improving readability. I am really glad I did this one, as it gave me an opportunity to put what I was studying before to good use. -
codelens-sample- This was a relatively straightforward refactor that came out well because the original was so horrible. While the rest of the samples I've looked at were competent (if clumsy in places), whoever wrote this was completely lost while doing it. -
call-hierarchy-sample- The second one I tried refactoring and I had to redesign quite a bit. I ended up doing quite a bit of work on this one to make it more functional. I removed whatever code duplication I could find and the result came out better than the original. The original has been doing a bunch of unecessary work that I've eliminated. -
contentprovider-sample- Unlike thequickinput-sampleI tackled this one with the intention of significantly cutting it down and failed at that, but at least I've managed to succeed at making it more readable.
For a blow by blow account what I've been doing here, see the Spiral repo commits going back two weeks from here.
By now I've worked roughly two weeks on this, and it has given me some first hand experience at using Typescript. I am a beginner at JS/TS, but I have significant experience programming in F# and my own language Spiral. So even though I am programming in a completely new language I am confident in my ability to see what is good code and what is not. When I say readability what I really mean is mental effort needed to understand the code one is studying. So between people, familiarity and their general level will determine the comfort zone of what they find readable.
As I was doing this I really missed: global type inference, expression based syntax and pattern matching. My view is that I definitely do not want to program in TS unless circumstances force it, but the language does have good points to it. The type system it uses is rather innovative, and it is a definite improvement on vanilla JS. A lot of what is bad in TS, it straight up inherited from JS.
Still, unlike the ML family of languages which have type systems worth emulating, I can tell that despite TS's excellent editor support that puts the rest of the web languages to shame, there is significant complexity in getting all of this to work under the hood.
This has to be praised, but a certain design point in TS sticks out in my mind. const foo (f : (arg: number) => number) = .... Functions are required to have variable names in their type signature. This is just nasty, and really cuddles the monkeys among the JS crowd a bit too much. The reason why it is so nasty is because naming is a really difficult problem, and having to do it for higher order function arguments really discourages style development in the correct direction.
JS could have been so much better if it's syntax wasn't filched from C. One thing I've noticed is that balancing parentheses becomes rather difficult once you try compressing an expression and putting it on the same line - this is something you'd want to do because of readability. Scrolling and flipping between pages takes mental energy, and so harms readability, but the language seems to actively fight the effort to do the right thing by having poor syntax.
It is a pity. I mean, JS did not conquer the world of programming by the merit of its technical excellence. Compared to the effort that has gone to make all this work under the hood, writing a parser is easy in comparison. So you'd expect that there would be more old languages adopting better styles, but that has not happened, instead looking at Bucklescipt to ReasonML transition, the move seems to be happening in the wrong direction. Still at some point, for the BS users at least - the notion that Ocaml's syntax is better should dawn on them.
The JS crowd is hopeless.
The number of good language designs and good language features is limited in amount, and I will do my own to converge on what I think is the best tradeoff. The difficult part of making a good language is due to the fact that a lot of features in the design space actually block other features. So you can do one thing to make your life easier only to have that make your life more difficult in many other areas. JS is a poster child for why language design takes care. It is the biggest wasted oppotunity in the world for doing good PL work.
I do not want to keep programming in TS, if I can help it. Continued usage of it will rot my brain.
I'll do my own part to create the next version of Spiral with the lessons of 2018 put to good use. The exercise I've done here has been done for the purpose of creating the tooling for it. All the pieces are in place, and I can finally start. Whatever I am missing I know I'll be able to pick up on the fly.
I am finally done with this. The last three months have been too long.
///
I wrote this in the readme for the fork of the samples repo.
Though I did good work in many places, only the quickinput-sample is really worth showing off. So there is no point in really opening an issue for this. It is possible the authors would get mad for fiddling with their 'perfectly good' samples, and I do not want the hassle if that happens. I can hold strong opinions, but I can't really go voicing them out loud too much. It will just seem like I am quibling about irrelevant stuff to the onlookers and forcing my style on others.
7:15pm. Mhhh...I had wanted to talk more about the samples themselves, but instead talked about TS instead and turned this into a more generic rant.
The last three months really have been too long. I can only blame my own ego and perfectionism for letting it come to this.
With the benefit of experience I know I could have cut it down much more significantly. In the end, I did not have to redo these samples.
Starting tomorrow, I will seriously start pursuing doing editor support for Spiral. The minimal viable product will not be too difficult. Still, it will be a lot of work that I really need to get started on. By the time the mid-2020 review comes around, I will still have nothing to show for. But by the end of 2020, I should have Spiral in a working state.
This next part will be much more fun than the things I did in the last 5 months. I will finally be breaking some novel ground and bringing Spiral up to its full potential.
I will show the world what a real language is. v0.09 were just the scouts. The main force will come after that."
changes king goat arena to use the portal landmark system (#49735)
About The Pull Request
Permanent portals no longer randomly select a portal to use as their exit, and instead stick with the first one they find, they're supposed to only have one linked portal anyways.
Adds a portal exit landmark, one-way portals use this to define their exit as opposed to the weird as hell and pointless portal deletion system I was using before, also nice for mapping since they have different icons so you can more easily tell which is which.
Adds One-Use portals, they're just one-way portals that delete after they're used, was requested for a reference off-station thing someone was making, probably also useful for things like mobs that die and spawn portals to an area, most likely will be used in ice moon.
Goat ruin was the only thing using the permanent portal system for ruins so far since VR hub got removed, so I just moved that over to use the landmarks. Why It's Good For The Game
The old code is garbage I wrote like a year ago because it was packaged with VR hub, and someone added copy-pasta at some point, absolutely terrible. This cuts down on a lot. Changelog
🆑 refactor: Permanent Portal fixes and One-Way Portal improvements for mapping. /🆑
Push farther on the K2 demo (which uses structs).
I ran into one significantly different requirement that structs surface which maps and other basic Data Model stuff didn't: in order to track "doneness" invariants across a whole structure, we need the final interation on the child NodeAssembler (meaning the call to either the "Assign*" or the "Done" methods, depending on whether that system was used recursively) to update the parent's state machine and let it know it's now correct for it to progress. (So far so good; that part is the same for basic maps and lists.) The trouble is... with maps, we could easily do this with one wrapper type; for structs, which may have different types per field, this is a lot less trivial!
There's now a commentstorm in the 'map_K2_T2.go' file exploring alternatives for solving this. As usual, there are tradeoffs, and no single obviously correct choice. (E.g., the fastest, most inliner-friendly choice involves significant bloat to the generated code size and binary, so that one's certainly not a slam dunk.) Building a large enough prototype to disambiguate which of identified solution paths for this will perform best is of course a nice idea, but the volume of code that has to manifest to provide a sufficiently realistic benchmarkable prototype is... well, not looking feasible; we'll be better off doing a first draft of all codegen with a randomly chosen solution, and then trying variations from there.
(Amusingly/headdeskingly, at some point I had it in my head that this call-up'ing situation was going to be one of the downsides unique to the alternate plans that kept the old builder interface plus "freezing" marks, and that this redesign jumped over it; alas, no, that was silly: both designs need this for similar reasons. (This design is still lighter in other ways, though.))
But! None of this is looking like show-stoppers, just Hard Choices. So, I'm going to table this -- and instead switch back to filling out the rest of the "ipldfree" next gen node kinds, and then, combined with the confidence we already have from those benchmarks in the previous few commits... start porting interface changes into core!
(And just... assume everything will go according to plan when it's time to come back to this codegen stuff we're tabling here. If it doesn't... that'll be fun.)
Oh! Also seen here: a slight improvement to the map assembler state machine tracking, now using an enum rather than bool. This is definitely a good idea and I'm going to lift it back to the other map implementations immediately, even if much of the rest of the poor 'map_K2_T2.go' file is sadly being left in a storm of todo's.
reverted zeal city models
see if this fixes random ass access violations might've fucked up their mat_config or some shit cause if this is the cause of em iunno what i did cause they don't crash in HH
game sucks
I have crippling anxiety ho god ho fuck
Updating so I can test this shit properly
i2c: rk3x: Increase wait timeout to 1 second
Although unlikely, it is remotely possible for an i2c command to need more than 200ms complete. Unlike smbus, i2c devices can clock stretch for an unspecified amount of time. The longest time I've seen specified for a device is 144ms (bq27541 battery gas), but one could imagine a device taking a bit slower. 1 second "ought to be enough for anyone."
The above is not the only justifcation for going above 200ms for a timeout, though. It turns out that if you've got a large number of printks going out to a serial console, interrupts on a CPU can be disabled for hundreds of milliseconds. That's not a great situation to be in to start with (maybe we should put a cap in vprintk_emit()) but it's pretty annoying to start seeing unexplained i2c timeouts.
Note that to understand why we can timeout when printk has interrupts disabled, you need to understand that on current Linux ARM kernels interrupts are routed to a single CPU in a multicore system. Thus, you can get:
- CPU1 is running rk3x_i2c_xfer()
- CPU0 calls vprintk_emit(), which disables all IRQs on CPU0.
- I2C interrupt is ready but is set to only run on CPU0, where IRQs are disabled.
- CPU1 timeout expires. I2C interrupt is still ready, but CPU0 is still sitting in the same vprintk_emit()
- CPU1 sees that no interrupt happened in 200ms, so timeout.
A normal system shouldn't see i2c timeouts anyway, so increasing the timeout should help people debugging without hurting other people excessively.
Signed-off-by: Doug Anderson dianders@chromium.org Tested-by: Caesar Wang wxt@rock-chips.com Acked-by: Uwe Kleine-König u.kleine-koenig@pengutronix.de Signed-off-by: Wolfram Sang wsa@the-dreams.de