Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AWS credentials required in the new AWS Discovery using SPI mode #29

Closed
thurber opened this issue Jun 7, 2017 · 11 comments
Closed

AWS credentials required in the new AWS Discovery using SPI mode #29

thurber opened this issue Jun 7, 2017 · 11 comments
Milestone

Comments

@thurber
Copy link

thurber commented Jun 7, 2017

The documentation implies that the "access-key", "secret-key", and "iam-role" properties are optional and that Hazelcast can fallback to using the role and credentials assigned to the EC2 instance. This would be ideal so that I do not need to hardcode these values into my hazelcast.xml.

However, when I try to run a hazelcast cluster without these properties, startup will fail with an error stating that "access-key" property is required, or that "access-key" property cannot be blank.

Is there a way I can omit these configurations, or are they actually required?

Thanks for your help!

@ghost ghost self-assigned this Jun 9, 2017
@ghost ghost added the Type: Defect label Jun 9, 2017
@ghost
Copy link

ghost commented Jun 9, 2017

Thanks @thurber, we are on it.

@miere
Copy link

miere commented Jun 10, 2017

I had to be honest, although Hazelcast offers a great architecture design to my application, it have turned my life into a nightmare. Most of time I really can't trust the documentation you guys write. Other times stuffs just stop to work...

@emrahkocaman emrahkocaman added this to the 2.0.2 milestone Jun 29, 2017
emrahkocaman pushed a commit that referenced this issue Jun 29, 2017
* fixes for hazelcast-aws Github issues #29 and #22

* fixes for checkstyle errors

* fixes for checkstyle errors

* extended strategy factory tests

* removed Configuration to AwsConfig

* Configuration fixes, we now allow w/out any creds, or iam-role defined in config file.

* missing default value for timeout added
@emrahkocaman
Copy link
Contributor

Documentation is correct, "access-key", "secret-key", "iam-role" are optional. This is a bug introduced while implementing Discovery SPI hazelcast-aws.
Fixed by #30 and will be available in 2.0.2 release. (2.0.2 should be released within 1-2 weeks).

@thurber Thanks for reporting, it'd be appreciated if you could give a try to 2.0.2-SNAPSHOT

@thurber
Copy link
Author

thurber commented Jun 29, 2017

Thanks for the update @emrahkocaman! Unfortunately my instances can no longer discover each other with the 2.0.2-SNAPSHOT provided above, regardless of including the settings or not (although I no longer receive errors for omitting them).

For reference, here's the relevant piece of my config xml that worked with 2.0.1:

...
<join>
      <tcp-ip enabled="false"></tcp-ip>
      <multicast enabled="false"/>
      <aws enabled="false" />
      <discovery-strategies>
        <discovery-strategy enabled="true" class="com.hazelcast.aws.AwsDiscoveryStrategy">
          <properties>
            <property name="access-key">not-needed</property>
            <property name="secret-key">not-needed</property>
            <property name="iam-role">aws-elasticbeanstalk-ec2-role</property>
            <property name="region">us-west-2</property>
            <property name="tag-key">my-tag-key</property>
            <property name="tag-value">my-tag-value</property>
          </properties>
        </discovery-strategy>
      </discovery-strategies>
    </join>
...

With 2.0.2-SNAPSHOT, I tried:

  • the same settings as above
  • omitting "access-key" and "secret-key" properties
  • omitting "access-key", "secret-key", and "iam-role" properties

I did not try:

  • using a real "access-key" and real "secret-key" (dummy values were fine with 2.0.1)

@ghost
Copy link

ghost commented Jun 30, 2017

@thurber Hi,

Could you please share hazelcast logs on each EC2 instances?

@mesutcelik mesutcelik reopened this Jun 30, 2017
@thurber
Copy link
Author

thurber commented Jun 30, 2017

Sure thing, see below for four log files -- two from instances in the working case with 2.0.1, and two from instances in the non-working case with 2.0.2-Snapshot. Let me know if I should set a more informative logging level. Also note that the application is running inside a docker container on each instance.

Non-Working Case (2.0.2-SNAPSHOT):
Instance 1:

Starting clustering... 
No cluster-host specified so using address 172.17.0.2 
[LOCAL] [dev] [3.8.2] Prefer IPv4 stack is true. 
[LOCAL] [dev] [3.8.2] Picked [172.17.0.2]:5701, using socket ServerSocket[addr=/0:0:0:0:0:0:0:0,localport=5701], bind any local is true 
[LOCAL] [dev] [3.8.2] Using public address: [172.31.43.137]:5701 
[172.31.43.137]:5701 [dev] [3.8.2] Hazelcast 3.8.2 (20170518 - a60f944) starting at [172.31.43.137]:5701 
[172.31.43.137]:5701 [dev] [3.8.2] Copyright (c) 2008-2016, Hazelcast, Inc. All Rights Reserved. 
[172.31.43.137]:5701 [dev] [3.8.2] Configured Hazelcast Serialization version : 1 
[172.31.43.137]:5701 [dev] [3.8.2] Backpressure is disabled 
[172.31.43.137]:5701 [dev] [3.8.2] Activating Discovery SPI Joiner 
[172.31.43.137]:5701 [dev] [3.8.2] Starting 2 partition threads 
[172.31.43.137]:5701 [dev] [3.8.2] Starting 3 generic threads (1 dedicated for priority tasks) 
[172.31.43.137]:5701 [dev] [3.8.2] [172.31.43.137]:5701 is STARTING 
[172.31.43.137]:5701 [dev] [3.8.2] Cluster version set to 3.8 
[172.31.43.137]:5701 [dev] [3.8.2] 

Members [1] {
	Member [172.31.43.137]:5701 - 0ad18a98-d972-4d8c-ab0d-8aad45fc2aeb this
}
 
[172.31.43.137]:5701 [dev] [3.8.2] [172.31.43.137]:5701 is STARTED 
[172.31.43.137]:5701 [dev] [3.8.2] Initializing cluster partition table arrangement...

Instance 2:

Starting clustering... 
No cluster-host specified so using address 172.17.0.2 
[LOCAL] [dev] [3.8.2] Prefer IPv4 stack is true. 
[LOCAL] [dev] [3.8.2] Picked [172.17.0.2]:5701, using socket ServerSocket[addr=/0:0:0:0:0:0:0:0,localport=5701], bind any local is true 
[LOCAL] [dev] [3.8.2] Using public address: [172.31.7.32]:5701 
[172.31.7.32]:5701 [dev] [3.8.2] Hazelcast 3.8.2 (20170518 - a60f944) starting at [172.31.7.32]:5701 
[172.31.7.32]:5701 [dev] [3.8.2] Copyright (c) 2008-2016, Hazelcast, Inc. All Rights Reserved. 
[172.31.7.32]:5701 [dev] [3.8.2] Configured Hazelcast Serialization version : 1 
[172.31.7.32]:5701 [dev] [3.8.2] Backpressure is disabled 
[172.31.7.32]:5701 [dev] [3.8.2] Activating Discovery SPI Joiner 
[172.31.7.32]:5701 [dev] [3.8.2] Starting 2 partition threads 
[172.31.7.32]:5701 [dev] [3.8.2] Starting 3 generic threads (1 dedicated for priority tasks) 
[172.31.7.32]:5701 [dev] [3.8.2] [172.31.7.32]:5701 is STARTING 
[172.31.7.32]:5701 [dev] [3.8.2] Cluster version set to 3.8 
[172.31.7.32]:5701 [dev] [3.8.2] 

Members [1] {
	Member [172.31.7.32]:5701 - 27a8ab7e-ce02-48cb-8480-484f180e433e this
}
 
[172.31.7.32]:5701 [dev] [3.8.2] [172.31.7.32]:5701 is STARTED 
[172.31.7.32]:5701 [dev] [3.8.2] Initializing cluster partition table arrangement...

Working Case (2.0.1):
Instance 1:

Starting clustering... 
No cluster-host specified so using address 172.17.0.2 
[LOCAL] [dev] [3.8.2] Prefer IPv4 stack is true. 
[LOCAL] [dev] [3.8.2] Picked [172.17.0.2]:5701, using socket ServerSocket[addr=/0:0:0:0:0:0:0:0,localport=5701], bind any local is true 
[LOCAL] [dev] [3.8.2] Using public address: [172.31.43.90]:5701 
[172.31.43.90]:5701 [dev] [3.8.2] Hazelcast 3.8.2 (20170518 - a60f944) starting at [172.31.43.90]:5701 
[172.31.43.90]:5701 [dev] [3.8.2] Copyright (c) 2008-2016, Hazelcast, Inc. All Rights Reserved. 
[172.31.43.90]:5701 [dev] [3.8.2] Configured Hazelcast Serialization version : 1 
[172.31.43.90]:5701 [dev] [3.8.2] Backpressure is disabled 
[172.31.43.90]:5701 [dev] [3.8.2] Activating Discovery SPI Joiner 
[172.31.43.90]:5701 [dev] [3.8.2] Starting 2 partition threads 
[172.31.43.90]:5701 [dev] [3.8.2] Starting 3 generic threads (1 dedicated for priority tasks) 
[172.31.43.90]:5701 [dev] [3.8.2] [172.31.43.90]:5701 is STARTING 
[172.31.43.90]:5701 [dev] [3.8.2] Connecting to /172.31.26.87:5701, timeout: 0, bind-any: true 
[172.31.43.90]:5701 [dev] [3.8.2] Connecting to /172.31.36.229:5701, timeout: 0, bind-any: true 
[172.31.43.90]:5701 [dev] [3.8.2] Could not connect to: /172.31.26.87:5701. Reason: SocketException[Connection refused to address /172.31.26.87:5701] 
[172.31.43.90]:5701 [dev] [3.8.2] [172.31.26.87]:5701 is added to the blacklist. 
[172.31.43.90]:5701 [dev] [3.8.2] Cluster version set to 3.8 
[172.31.43.90]:5701 [dev] [3.8.2] 

Members [1] {
	Member [172.31.43.90]:5701 - 06259fb9-e61f-43e0-a58b-5794c6a7a55c this
}
 
[172.31.43.90]:5701 [dev] [3.8.2] [172.31.43.90]:5701 is STARTED 
[172.31.43.90]:5701 [dev] [3.8.2] Initializing cluster partition table arrangement... 
Failed to find a usable hardware address from the network interfaces; using random bytes: fd:ac:6b:36:f4:f6:30:92 
[172.31.43.90]:5701 [dev] [3.8.2] Accepting socket connection from /172.17.0.1:36512 
[172.31.43.90]:5701 [dev] [3.8.2] Established socket connection between /172.17.0.2:5701 and /172.17.0.1:36512 
[172.31.43.90]:5701 [dev] [3.8.2] 

Members [2] {
	Member [172.31.43.90]:5701 - 06259fb9-e61f-43e0-a58b-5794c6a7a55c this
	Member [172.31.26.87]:5701 - abfdefc6-a98c-468c-aa0e-fb715d83358d
}
 
[172.31.43.90]:5701 [dev] [3.8.2] Re-partitioning cluster data... Migration queue size: 271

Instance 2

Starting clustering... 
No cluster-host specified so using address 172.17.0.2 
[LOCAL] [dev] [3.8.2] Prefer IPv4 stack is true. 
[LOCAL] [dev] [3.8.2] Picked [172.17.0.2]:5701, using socket ServerSocket[addr=/0:0:0:0:0:0:0:0,localport=5701], bind any local is true 
[LOCAL] [dev] [3.8.2] Using public address: [172.31.26.87]:5701 
[172.31.26.87]:5701 [dev] [3.8.2] Hazelcast 3.8.2 (20170518 - a60f944) starting at [172.31.26.87]:5701 
[172.31.26.87]:5701 [dev] [3.8.2] Copyright (c) 2008-2016, Hazelcast, Inc. All Rights Reserved. 
[172.31.26.87]:5701 [dev] [3.8.2] Configured Hazelcast Serialization version : 1 
[172.31.26.87]:5701 [dev] [3.8.2] Backpressure is disabled 
[172.31.26.87]:5701 [dev] [3.8.2] Activating Discovery SPI Joiner 
[172.31.26.87]:5701 [dev] [3.8.2] Starting 2 partition threads 
[172.31.26.87]:5701 [dev] [3.8.2] Starting 3 generic threads (1 dedicated for priority tasks) 
[172.31.26.87]:5701 [dev] [3.8.2] [172.31.26.87]:5701 is STARTING 
[172.31.26.87]:5701 [dev] [3.8.2] Connecting to /172.31.36.229:5701, timeout: 0, bind-any: true 
[172.31.26.87]:5701 [dev] [3.8.2] Connecting to /172.31.43.90:5701, timeout: 0, bind-any: true 
[172.31.26.87]:5701 [dev] [3.8.2] Established socket connection between /172.17.0.2:46713 and /172.31.43.90:5701 
[172.31.26.87]:5701 [dev] [3.8.2] Cluster version set to 3.8 
[172.31.26.87]:5701 [dev] [3.8.2] 

Members [2] {
	Member [172.31.43.90]:5701 - 06259fb9-e61f-43e0-a58b-5794c6a7a55c
	Member [172.31.26.87]:5701 - abfdefc6-a98c-468c-aa0e-fb715d83358d this
}
 
[172.31.26.87]:5701 [dev] [3.8.2] [172.31.26.87]:5701 is STARTED

@ghost
Copy link

ghost commented Jul 3, 2017

@thurber Hi,

Is there any specific reason to use directly Docker on EC2 host rather than AWS ECS?

Moreover, is it possible to check below items on your environment?
1] Please run docker containers with --network=host networking on EC2 hosts.
2] Please add below interface binding to hazelcast.xml configuration file.

<interfaces enabled="true">
        <interface>10.0.*.*</interface>
</interfaces>

Please note that 10.0.*.* depends on your subnet or CIDR block.

3] Please make sure that 2.0.2-SNAPSHOT is accessible with cp or CLASSPATH, there should be similar log entries when you start hazelcast instance

WARNING: Describe instances will be queried with iam-role assigned to EC2 instance, please make sure given iam-role have ec2:DescribeInstances policy attached.

or you can check here

please let us know, if further help or support is required. thanks in advance.

@thurber
Copy link
Author

thurber commented Jul 3, 2017

@lazerion Ack sorry about that, noob mistake: I wasn't setting the snapshot jar into the classpath correctly. It works like a charm now, thanks!

I am able to avoid needing --network=host and the <interface> option by specifying the <network><public-address> option programmatically before launch.

@destebanm
Copy link

Hi!

When do you plan to release 2.0.2?

Thanks!

@mesutcelik
Copy link
Contributor

Hi @destebanm ,
We are working on ZONE_AWARE feature where hazelcast partition groups will be defined based on AWS Availability Zones. We are planning to release in a few weeks and the version will be 2.1

@destebanm
Copy link

Thanks for the info @mesutcelik

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants