Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added home, data, config paths #1373

Merged
merged 2 commits into from
Apr 13, 2016
Merged

Added home, data, config paths #1373

merged 2 commits into from
Apr 13, 2016

Conversation

tsg
Copy link
Contributor

@tsg tsg commented Apr 11, 2016

Implements part of #1371.

  • Adds the CLI flags for 'path.home', 'path.data', 'path.config'
  • These can be also set in the configuration file:
path:
  home: /tmp/
  data: /tmp/data/
  config: /tmp/config/
  • Filebeat registry location is now by default {data_path}/registry. Note: this is
    a breaking change from .filebeat previously in 1.X.
  • The ES templates are searched by default in the config path

@tsg tsg mentioned this pull request Apr 11, 2016
14 tasks
@ruflin
Copy link
Member

ruflin commented Apr 11, 2016

Thinking about migration. Should we have a script that checks for the old registry file on startup and moves it to the new place in 5.0?

@tsg
Copy link
Contributor Author

tsg commented Apr 11, 2016

For the non-deb/rpm case, the old location is anyone's guess. It was .filebeat in the working directory, so it depends on how filebeat was executed. For the deb/rpm case, BC is maintained. I hope most people used deb/rpm.

@tsg tsg added the review label Apr 12, 2016
@tsg
Copy link
Contributor Author

tsg commented Apr 12, 2016

This is now ready for reviews. The rest of the items #1371 will be done in other PRs.

@@ -14,7 +14,7 @@ import (

// Defaults for config variables which are not set
const (
DefaultRegistryFile = ".filebeat"
DefaultRegistryFile = "registry"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should add the .json extension to the name.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason I used registry is that on DEB/RPM installations the file name already used is /usr/share/filebeat/registry. By using registry we can keep BC for those cases at least.

In addition, JSON is an implementation detail which we might change in the future. Also, adding the extension could be inviting people to edit the file, which I guess is the last thing we want.

I see the three +1, but is there a strong reason to add the .json?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed my +1 as I think the point about implementation and potentially changing in the future is a good point. So even if we switch to the format "xyz" the file name stays the same.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't have a strong opinion here. So leaving it as registry is fine with me. (I never meant to +1 my own comment anyways; I didn't realize clicking the thumb icon added a +1 until after I clicked it, and then I couldn't figure out how to undo.)

The reason I used registry is that on DEB/RPM installations the file name already used is /usr/share/filebeat/registry. By using registry we can keep BC for those cases at least.

Those installations were using a non-default value so they should not be affected by the change.

In addition, JSON is an implementation detail which we might change in the future.

True, leaving it without an extension makes it easier to change underlying file type.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Those installations were using a non-default value so they should not be affected by the change.

Good point. But if it's just registry I can get rid of one of those nasty sed commands on the configuration file at package time by just setting the data path in the init script.

@andrewkroh
Copy link
Member

It would be good to do a search for .filebeat in all files and update any references to the new registry file name.

@tsg
Copy link
Contributor Author

tsg commented Apr 12, 2016

I replaced all occurrences of .filebeat with the commit above, and in the process updated the docs a little. We'll need a "guide" on the paths, though, similar with what ES has. That will come in a separate PR (cc @dedemorton, just FYI).

@@ -390,13 +390,13 @@ filebeat:

===== registry_file

The name of the registry file. By default, the registry file is put in the current
working directory. If the working directory changes for subsequent runs of Filebeat, indexing starts from the beginning again.
The name of the registry file. If a relative path is used, it is considered relative to the
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure what you mean by data path here. Do you mean relative to the folders that are being crawled? Does that mean there is more than one registry file?

Tudor Golubenco added 2 commits April 13, 2016 09:49
* Adds the CLI flags for 'path.home', 'path.data', 'path.config'
* These can be also set in the configuration file:

```
path:
  home: /tmp/
  data: /tmp/data/
  config: /tmp/config/
```

* Filebeat registry location is now by default `{data_path}/registry`. Note: this is
a breaking change from `.filebeat` previously in 1.X.
* The ES templates are searched by default in the config path
@tsg
Copy link
Contributor Author

tsg commented Apr 13, 2016

Cleaned history and rebased to master.

@andrewkroh
Copy link
Member

LGTM

@urso urso merged commit 56a0296 into elastic:master Apr 13, 2016
@tsg tsg deleted the directory_layout branch August 25, 2016 10:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants